I was recently working on a multi-vector dataset comparison in Deep Space. We had received Revit, Navisworks and tabular data, and I was comparing 3 different data drops of that information, particularly for changes in the quantities of specific types of elements. A ‘data drop’ is a set of data or files that you receive at a given point in time.

In the course of exporting the Navisworks data through to CSV, I came across a specific problem – the number of rows in imported CSV did not match the number of elements processed. Why?

As you may know, you can store line breaks inside parameters in Revit. There are very few good reasons to do this, but it still does happen. Once this happens, those line breaks need to be processed by tools down stream. I spent a lot of time in the early days of Deep Space figuring out how to ‘clean’ bad Revit data so we could still bring it into the platform for analysis. Usually, if the CSV writer or reader are smart enough, they should be able to deal with this problem. However, I discovered a bit of a gap in the available tools this time. What was the problem?

It turns out that the CSV was malformed, it was actually dirty or bad data. While it did have the line breaks, it did not consistently use double quotes to contain fields. So we had the situations where there would be line breaks that were inside the CSV fields or columns, but not inside double quotes. I tried a lot of different CSV readers, including Excel, LibreOffice, Google Sheets and PowerBI / PowerQuery, but they all tripped up at this data. Because they were using the rule ‘new line = new row of data’, the imported CSV information was coming in mangled.

How can we clean this data? We generally know there should be 17 fields or 16 commas per row of data. But we also know there can be line breaks inside fields… so it is a challenge to map a data row to CSV lines. In some cases 2 or 3 lines of CSV data might still just be one row of actual data.

After trying to use various out of the box solutions, I decided to build some Python code to try and solve this. I used Dynamo Sandbox 1.3 to do this, primarily out of habit, not because it is the best Python IDE out there 🙂 I ended up with a kind of line-merging iterator, here is some of the Python code below:

biglist = IN[0]

commact=[]
for ctr in range(len(biglist)):
    astr=biglist[ctr]
    strct=astr.count(',')
    commact.append(strct)

counted=range(len(biglist))
fixedstr=[]
bad=[]
skips=[]
incr=0

for ctrx in range(len(biglist)):
    if ctrx==skips:
        pass
    elif commact[ctrx]==16:
        fixedstr.append(biglist[ctrx])
    elif commact[ctrx]>16:
        fixedstr.append(biglist[ctrx])
    elif commact[ctrx]+commact[ctrx+1]==16:
        jnr=[]
        jns=biglist[ctrx]+biglist[ctrx+1]
        fixedstr.append(jns)
        skips=ctrx+1
    else:
        bad.append(ctrx) 
        
OUT = bad, fixedstr, commact, counted

What does it do? Essentially, it counts the number of delimeters (commas) on one line, then it processes or merges lines based on that information.

This actually got me around 90% of the way there. Then I still had to do some manual fixes of things like ‘double double quotes’ that were also tripping up the CSV readers.

Above is a snapshot of the Dynamo script. And here is the script for download:

CSV Line Merge for Malformed CSV

What the the lessons here?

  1. Try and fix the source or native data if you can. Dealing with messy data downstream can be a real pain.
  2. If you need to solve this problem, you can pick up my code or work above and advance it a bit further to build a more robust ‘malformed CSV reader’
  3. Don’t let problems like this distract you during the holidays 🙂

As software and standards mature and even become archaic, they inevitably attract baggage along the way. Years of technical debt amassed by well-intentioned developers and product managers will be paid for by us and our children.

This is particularly evident when we start talking about the Id of Revit elements, and the IFC GUID syntax. As most of you are aware, Revit carries a number of different identifying attributes for each and every element. Here is a basic list:

  • Element Id – the numerical form of a Revit Id, you can interact directly with this using the Select By Id command in Revit
  • UniqueId – “A stable unique identifier for an element within the document.” This is not a correctly formed UUID, but rather a concatenation of a standard UUID for the Revit EpisodeId (session based), along with the Revit element Id. From Jeremy’s post: similar to the standard GUID format, but has 8 additional characters at the end. These 8 additional hexadecimal characters are large enough to store 4 bytes or a 32 bit number, which is exactly the size of a Revit element id.
  • DwfGuid (Export Guid) – this is a correctly formed UUID in .NET / standard format
  • IfcGuid – identical to DwfGuid, but encoded differently according to IFC rules created in the mid-90s. At the time, it was deemed worthwhile to ‘compress’ the IFC Guid to the shorter form we still have to deal with today.

It would be nice if Revit and IFC shared a single common unique identifying syntax, but unfortunately this is not the case.

The above Ids do actually bear some predictable relationship to each other. The UniqueId is formed by Revit joining the EpisodeId and the encoded Element Id, the Dwf or Export Guid is created by an algorithm that has a number of conditional rules, and the Dwf Guid can be converted to the backwards-compatible IfcGuid format through a different algorithm, originally created in C. Those algorithms can be sourced in various places (see links in Further Reading below).

Also, some of these get exposed in different ways – if you create Element-bound BCF files, they will typically show the IFC Guid syntax. Further, if you export an IFC file from Revit and tick the option on the Exporter, it will write the IfcGuid to a parameter on the element.

You can query the Element Id directly in Revit (Modify ribbon, Inquiry panel, IDs of Selection):

However, for the purpose of this post, let’s assume you are using Dynamo with IronPython, and you want to query all 4 Ids from an element.

We at least need to import the Revit API and the Revit IFC API:

clr.AddReference("RevitAPI")
import Autodesk 
from Autodesk.Revit.DB import *
clr.AddReference('RevitAPIIFC')
from Autodesk.Revit.DB.IFC import *

 

Following this, we can use the various Dynamo and Python commands to access the Ids:

elementIds, uniqueIds, DwfGuids, IfcGuids, successlist = [], [], [], [], []
for i in e:
    try:
        elementIds.append(i.Id)
        uniqueIds.append(i.UniqueId)
        DwfGuids.append(ExportUtils.GetExportId(doc, ElementId(i.Id)))
        IfcGuids.append(ExporterIFCUtils.CreateSubElementGUID (UnwrapElement(i),0))
        successlist.append("Success")
    except:
        successlist.append("Failure")
OUT = elementIds, uniqueIds, DwfGuids, IfcGuids, successlist

 

Notice the commands and properties used:

  • Element Id – query the Element.Id member
  • Unique Id – query the Element.UniqueId member
  • Dwf or Export Guid – use the ExportUtils.GetExportId method
  • IfcGuid – use the ExporterIFCUtils.CreateSubElementGUID method (index 0 refers to the main element itself)

 

From here, I create a Dynamo node that eats Revit elements and gives us back these Id values:

before node created

 

After “All The Ids” node created

 

This node will be published in the Bakery package on the package manager, and to Github.

 

Further, our VirtualBuiltApp platform has been developed to store and query multiple Ids for a single element.

 

Example output from the Dynamo / Python (initial test showed the Dwf Guid is still a .NET object, that probably should get converted to a string for consistency).

[2424]
['c98618bf-7112-4e90-8a71-8ab768f2b1c0-00000978']
[<System.Guid object at 0x0000000000000071 [c98618bf-7112-4e90-8a71-8ab768f2b8b8]>]
['39XXY$SH9Ea8fnYhTeyhYu']

 

I added str() to the Python:

 

Final test showing the 4 different Id values for a single object:

2424
'c98618bf-7112-4e90-8a71-8ab768f2b1c0-00000978'
'c98618bf-7112-4e90-8a71-8ab768f2b8b8'
'39XXY$SH9Ea8fnYhTeyhYu'

 

Further reading:

I came across this link to a Dynamo seminar by Sol Amour delivered in Wellington about a month ago. I have had a bit of contact with Sol over the years and he is a Dynamo pro. Cool to see that Dynamo Nodes got mentioned too.

Check out the seminar here.

As you can tell by his headshot, he means business 🙂

There is some more information about the event at this link.

Interesting little release by Dimitar Venkov on Github a few months ago. It is essentially a Python shell for Navisworks 2016. You install by unzipping as per instructions below. You may have heard about RevitPythonShell, but obviously this one is for Navis.

To install, simply extract the zip archive in the below folder:

%APPDATA%\Autodesk Navisworks Manage 2016\Plugins

Downloads

You can read more about the features at the main page here.

  • interactive IronPython interpreter for exploring the API
    • with syntax highlighting and autocompletion (in the console only)
    • based on the IronLab project
  • batteries included! (Python standard library is bundled as a resource in the RpsRuntime.dll)
  • full access to the .NET framework and the Navis API
  • configurable “environment” variables that can be used in your scripts
  • save “external scripts” for reuse and start collecting your awesome hacks!
  • run scripts at Navisworks startup

And some example/s are in a GitHub folder:

The Revit API is actually something pretty special. People will go on and on about how Revit needs this feature or that feature, but the fact is that you can build almost any feature you like with the API. Recently, I have been running quite a few batch operations from the scope of a federated Revit model: so I will have one RVT file, with hundreds of Revit links, and I will process them from that main federated model.

On one recent project, we had to deliver to a Client a linked dataset, with Revit link file paths resolving correctly. As you know, people work in many different IT environments, and the pathing of Revit links may vary widely.

I set up an ‘approved’ list of Revit file paths, that looked something like this:

I knew that in Dynamo with Python I could get a lot of information about linked files using the ExternalFileReference class. What I discovered during this process is that there is a TransmissionData API class that let’s you do some pretty interesting things…

You see, I was thinking I would have to set up a batch method to open this files, change the file paths, and close them. But the TransmissionData class is basically what is implemented in eTransmit for Revit – it allows you to ‘lightly touch’ the Revit file and simply change the Revit link paths, and also set a switch saying ‘this file has been transmitted’. This puts the file in an appropriate state for re-opening in the new path environment. Pretty cool huh?

Once I figured out how to implement those TransmissionData actions in Python, I just had to build a node that, running from the federated model:

  • examines each link for the links inside of it
  • replaces erroneous paths with the correct file path
  • sets the new paths to the file

I did this in the hacky way of a “counter with List.Map” in Dynamo. In the future I’ll probably fix it up to be a ‘proper’ Python script but this works for now. In about an hour it fixed the linked file paths of 600 Revit links, all with the click of a single button 🙂

You can download the main definition here:

External References FINAL

You can get the supporting nodes from GitHub here:

https://github.com/LukeyJohnson/BakeryForDynamo/tree/master/nodes

As usual, please use with care. And it is probably worth backing up your files before running something like this.

It is kinda more Python than Dynamo but hey, you get the picture 🙂

In fact, here is the Python code:

import clr
clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *

# Import RevitAPI
clr.AddReference("RevitAPI")
import Autodesk
from Autodesk.Revit.DB import *

clr.AddReference("RevitServices")
import RevitServices
from RevitServices.Persistence import DocumentManager
from RevitServices.Transactions import TransactionManager

from System.Collections.Generic import *

clr.AddReference('RevitNodes')
import Revit
clr.ImportExtensions(Revit.Elements)
clr.ImportExtensions(Revit.GeometryConversion)

from System import Guid

import System

import sys
pyt_path = r'C:\Program Files (x86)\IronPython 2.7\Lib'
sys.path.append(pyt_path)

import os.path

doc = DocumentManager.Instance.CurrentDBDocument
uiapp = DocumentManager.Instance.CurrentUIApplication
app = uiapp.Application

tempvalue = IN[0]
approvedFilenames = IN[1]
approvedFilepaths = IN[2]
targetfilepath = IN[5]

def stripquotes(string):
	string = string[1:-1]
	return string

transData = TransmissionData.ReadTransmissionData(tempvalue)
erefids = transData.GetAllExternalFileReferenceIds()
refdata = []
for x in erefids:
	refdata.append(transData.GetDesiredReferenceData(x))

currentpaths, currenterefType, currenterefPath, pstr = [], [], [], []

for e in refdata:
	currentpaths.append(ExternalFileReference.GetAbsolutePath(e))
	currenterefType.append(e.ExternalFileReferenceType)
	currenterefPath.append(e.PathType)
for s in currentpaths:
	pstr.append(ModelPathUtils.ConvertModelPathToUserVisiblePath(s))
	
filenames = []
for p in pstr:
	templist = os.path.split(p)
	filenames.append(templist[1])
	
newpath = []
indices = []
failpath = []
origcounter = 0
matchrefs = []
newpathtypes, newbools = [], []
pathtypevar = IN[3]
for f in filenames:
	tempindex = approvedFilenames.index(f) if f in approvedFilenames else -1
	indices.append(tempindex)
	if tempindex == -1:
		failpath.append(origcounter)
		pass
	else:
		newpath.append(ModelPathUtils.ConvertUserVisiblePathToModelPath(stripquotes(approvedFilepaths[tempindex])))
		matchrefs.append(erefids[origcounter])
		newpathtypes.append(pathtypevar)
		newbools.append(True)
	origcounter = origcounter + 1

currentfilepathstring=ModelPathUtils.ConvertModelPathToUserVisiblePath(targetfilepath)
elementcount = len(erefids)
hostfile = currentfilepathstring * elementcount
currentdata = []
#currentdata.append(transData)
currentdata.append(hostfile)
currentdata.append(erefids)
currentdata.append(refdata)
currentdata.append(currentpaths)
currentdata.append(currenterefType)
currentdata.append(currenterefPath)
currentdata.append(pstr)
currentdata.append(filenames)

newdata = []
newdata.append(indices)
newdata.append(newpath)
newdata.append(matchrefs)
newdata.append(newpathtypes)
newdata.append(newbools)

setlength = len(newpath)
setcounter = range(setlength)
successreport = []
setdata = IN[4]
if setdata:
	for s in setcounter:
		try:
			transData.SetDesiredReferenceData(matchrefs[s], newpath[s], newpathtypes[s], newbools[s])
			successreport.append("Success setting data")
		except:
			successreport.append("Failure setting data")
else:
	successreport.append("You need to set the switch to True")

if setdata:
	try:
		transData.IsTransmitted = True
		transData.WriteTransmissionData(targetfilepath, transData)
		successreport.append("Success WRITING data")
	except:
		successreport.append("Failure WRITING data")
		
#Assign your output to the OUT variable.
OUT = successreport, currentdata, newdata
 

If you want to read more about the API methods used:

TransmissionData

ModelPathUtils

ExternalFileReference

Have you ever wondered how to convert a Revit macro into an addin? You can follow Harry’s steps over at this post.

What about getting some sample code for Python and Ruby Revit projects? From AEC DevBlog:

… the code examples are hidden in the default macro projects created by macro manager, see below image, the project is the default python project, there are several “if False” statements, if you remove the line of “if False”, uncomment and unindent 4 spaces to methods below it, the methods will become available macros and appear in the macro manager, and you will see the same “if false” in Ruby projects too:

How to find python macro exmaples

Original post: http://adndevblog.typepad.com/aec/2016/03/revitapi-how-to-find-code-examples-for-macro-of-python-or-ruby.html

You may also be interested in the Revit Python Wrapper here http://revitpythonwrapper.readthedocs.io/en/latest/

Konrad and Mostapha are working on something, and the functionality does not look shrimpy at all. Check it out:

“Mantis Shrimp is a Dynamo (Revit) and Grasshopper (Rhino) interoperability project that allows you to read Rhino’s native *.3dm file type as well as export geometry from Grasshopper. It is written in Python in form of a user objects (on Grasshopper side for exporting) and custom Python nodes (on Dynamo side for importing). It’s an OPEN SOURCE project with all of the source code available on GitHub. At the moment it’s a collaboration project between myself and Mostapha Sadeghipour.

I decided to make this project an open source for multitude of reasons but most importantly because it was written on top of Dynamo (an open source project) using OpenNurbs (an open source project) and inspired by Rhynamo (an open source project to be in December 2014), and finally I was helped along the way by Mostapha who’s almost never written anything that he didn’t like to share. I think i got the “bug” – not Ladybug – for sharing from him.

Here’s how to get started with Mantis Shrimp…”

Read the rest at:
mantis shrimp – getting started… | archi-lab

Check out the solution suggested by Andreas Dieckmann:

“I ended up using one of Python’s abilities to handle lists which is called groupby() because I did not want my head to hurt thinking about how to go about this in Dynamo… 😉
Here’s how:
  1. Sort list by X values using the method Steve explains above
  2. Sort list by Z values 
    (Notice that within each group of equal Z values, the X values retain their order from the previous sort operation)
  3. Create a list of lists with XYZ values (sublist 0) and just Z values (sublist 1)
  4. Group that list of lists by the Z values (using a custom node with some Python code inside)
  5. Extract only the list of lists of XYZs by combining two map nodes
You will need the package Group List of Lists By Key for this to work.
(For what it’s worth, I also uploaded another package called Sort List Of Lists this week which was related to that project but isn’t needed here. But it might be worth checking out, too.)”

Heads-up:
https://twitter.com/Jbenoit44/status/420673580838293504