A Singleton is Deallocated

In the previous posts in this series the Parsers have used a hypothetical XMLParsableType Protocol for fetching data from an XML Element. I previously mentioned that there are myriad libraries and methods for parsing an XML document for iOS. All of the code in this article and covered in previous articles is available on GitHub.

In this post we’re going to take a look at two things: 1. How XMLParsableType can be implemented using libxml2. 2. How the various interfaces of libxml2 and abstractions in Swift can result in drastically variant performance.

Parser Interfaces

The immensely popular Open Source project libxml2 is a C Framework bundled with iOS and Mac OS X for parsing XML¹, with no external dependencies. For our interests it has implementations for a number of XML Reading interfaces. These are the DOM, SAX and Reader interfaces.

If I were to build these interfaces in an Application I’d want to be cautious to not prematurely optimise the XML Document reading process. Even if there are any substantial gains to XML Parsing performance when building implementations of the XMLParsableType, the gains may not be that significant to the Application as a whole. If the implementations were to be made from scratch it would be worth writing one simple implementation and the swapping it with a more performant implementation if-and-when it is seen to be impacting the performance of the Application at large. However, this is a fun exercise to play with Swift and Xcode 6 as well as getting to grips with the libxml2 library.

DOM

A DOM (Document Object Model) or Tree Interface to XML represents all of the Elements within an XML Document as a tree of connected nodes. This often means that the whole of an XML Document is read into memory and is navigable by following relationships in a data structure. By exposing the XML Document as a fully realized data structure, a DOM Interface very convenient to navigate.

By loading all parts of the XML Document into memory the memory usage is proportional to the size of the XML Document. This will be a low cost for small documents, but for larger documents this can result in a substantial amount of memory to allocate².

The performance of a DOM parser relative to others is largely dependent on the number of times that the same DOM is traversed and the amount of ignored data in the DOM. If the DOM is repeatedly traversed, then the up-front cost of building the tree will amortise later navigation of the data structure. However, if there are only a few Elements and Text nodes that need to be extracted from a large document, the resources used to read the unread parts of the tree are wasted.

SAX

As a DOM interface requires that the whole Tree of Elements is in memory an interface to XML exists that streams data out of the document on a per-node basis. The SAX interface traverses all of the nodes in sequence, allowing data to be extracted from a stream of callbacks. This means that the entirety of the document doesn’t have to be in memory in order to read a document. This can be very desirable in resource constrained environments, or when the document to be parsed is very large.

Unfortunately, this comes with drawbacks. The immediate concern for this article is that a callback based interface can be inconvenient as it results in temporary state being created for book-keeping purposes. A data-structure approach may be able to store all of the bookkeeping information in local variables and function arguments, resulting in simpler implementations that use the recursion and the call stack instead of allocated data structures.

Reader

The Reader Interface is very similar to the SAX interface, with a cursor based way of traversing the tree, rather than callbacks. This can be thought of as the difference between a push and pull driven stream of data. This makes the interface significantly easier to work with than SAX.

Swift Implementation

With a little context to the types of parsing interfaces that are available in libxml2, it help us to understand how they can be used in Swift. All of the implementations of the interface will return a Result<XMLParsableType> of the root node in the XML Document, with the XML Document read from either a File or an NSData representation of document String.

C Structures in Swift

libxml2 Open Source implementation of is a C library, the C functions can be imported into Swift using an Bridging Header and used as if they are Swift functions with usual rules of interacting with C API. As Working with the structures requires quite a bit of unwrapping and manipulating ‘Unsafe’ containers that wrap all C structures in Swift. Again, a strong reason to move up to the Swift level of abstraction.

Direct access of C structure members is not permitted in Swift as Swift does not know about the shape of structures. Any access of structure members will need to be wrapped in C/Objective-C functions and imported in the Bridging Header.

For example, this function will extract the content member from a Node:

NSString * LibXMLDOMGetText(xmlNodePtr node)
{
    NSCParameterAssert(node != NULL);
    NSCParameterAssert(node->type == XML_TEXT_NODE);

    const char *name = (const char *) node->content;
    NSCParameterAssert(name != NULL);

    return [NSString stringWithCString:name encoding:NSUTF8StringEncoding];
}

These functions can then be used in Swift when they are exposed in an Umbrella or Bridging Header.

GOTO Swift

I like to follow a principle when developing software, regardless of the languages and frameworks used:

“Get to the highest level of Abstraction that makes sense, as early as possible. Don’t think about the overhead of the abstraction until you can prove that it is detrimental to performance”

libxml2 sits very far down the ladder of Abstraction, so I consider it to be ideal to pull data from libxml2 into a higher level of abstraction as soon as possible. This makes manipulating and reading XML data in Swift significantly easier. Additionally, this means skipping Objective-C³ and using elements of the Swift Standard Library along with bridged Cocoa libraries to get stuff done. Swift provides us with some abstractions that are superior to the Objective-C equivalents, there’s no need for an intermediate stage unless it is needed. The Strong guarantees that Swift provides us are also present in the components of the Standard Library.

Conveniently, Xcode will parse headers with enumerations declared with the NS_ENUM macro:

typedef NS_ENUM(NSInteger, LibXMLReaderType) {
    LibXMLReaderTypeNONE = XML_READER_TYPE_NONE,
    LibXMLReaderTypeELEMENT = XML_READER_TYPE_ELEMENT,
    LibXMLReaderTypeATTRIBUTE = XML_READER_TYPE_ATTRIBUTE,
    LibXMLReaderTypeTEXT = XML_READER_TYPE_TEXT,
    LibXMLReaderTypeCDATA = XML_READER_TYPE_CDATA,
    LibXMLReaderTypeENTITY_REFERENCE = XML_READER_TYPE_ENTITY_REFERENCE,
    LibXMLReaderTypeENTITY = XML_READER_TYPE_ENTITY,
    LibXMLReaderTypePROCESSING_INSTRUCTION = XML_READER_TYPE_PROCESSING_INSTRUCTION,
    LibXMLReaderTypeCOMMENT = XML_READER_TYPE_COMMENT,
    LibXMLReaderTypeDOCUMENT = XML_READER_TYPE_DOCUMENT,
    LibXMLReaderTypeDOCUMENT_TYPE = XML_READER_TYPE_DOCUMENT_TYPE,
    LibXMLReaderTypeDOCUMENT_FRAGMENT = XML_READER_TYPE_DOCUMENT_FRAGMENT,
    LibXMLReaderTypeNOTATION = XML_READER_TYPE_NOTATION,
    LibXMLReaderTypeWHITESPACE = XML_READER_TYPE_WHITESPACE,
    LibXMLReaderTypeSIGNIFICANT_WHITESPACE = XML_READER_TYPE_SIGNIFICANT_WHITESPACE,
    LibXMLReaderTypeEND_ELEMENT = XML_READER_TYPE_END_ELEMENT,
    LibXMLReaderTypeEND_ENTITY = XML_READER_TYPE_END_ENTITY,
    LibXMLReaderTypeXML_DECLARATION = XML_READER_TYPE_XML_DECLARATION
};

This is a redeclaration of libxml enum values, with a naming convention that will play nice with Swift:

switch LibXMLDOMGetElementType(child) {
case .ELEMENT_NODE:
    children.append(LibXMLNodeParserDOM.createTreeRecursive(child))
case .TEXT_NODE:
    text = LibXMLDOMGetText(child)
default:
    break
}

Swift provides the SequenceType and GeneratorType protocols to allow us to use the for ... in ... syntax, or use the higher-level map, find and filter functions. We can define a Sequence type that exposes the Child nodes of an XML Node:

typealias LibXMLDOMNodeSequence = SequenceOf<xmlNodePtr>

internal class func childrenOfNode(node: xmlNodePtr) -> LibXMLDOMNodeSequence {
    var nextNode = LibXMLDOMGetChildren(node)
    let generator: GeneratorOf<xmlNodePtr> = GeneratorOf {
        if (nextNode == nil) {
            return nil
        }

        let currentNode = nextNode
        nextNode = LibXMLDOMGetSibling(nextNode) 
        return currentNode
    }

    return SequenceOf(generator)
}

By moving enumeration to a Sequence type, consumers of the DOM don’t need to be concerned with how the list of Child Nodes is navigated.

Implementation 1: Swift Structure from libxml2 DOM

The first thing that springs to mind is that given a DOM in libxml, some of the data can be pulled out into a pure Swift data structure⁴. This Swift data structure will contain all of the relationships and Text Nodes, whilst implementing the previously defined XMLParsableType.

public struct XMLNode: XMLParsableType {
    public let name: String
    public let text: String?
    public let children: [XMLNode]

    public func parseChildren(elementName: String) -> [XMLNode] {
        return self.children.filter { node in
            return node.name == elementName
        }
    }

    public func parseText() -> String? {
        return self.text
    }
}

The childrenOfNode function is placed in the LibXMLDOM class to separate the concerns of how the Tree is built into a data structure and how the DOM is navigated. By using this function that returns a Sequence`` and a number of other wrapper functions (prefixed withLibXMLDOM“`) returning Swift types, the construction of the tree can be defined recursively:

private class func createTreeRecursive(node: xmlNodePtr) -> XMLNode {
    let name: String! = LibXMLDOMGetName(node)
    var text: String?

    var children: [XMLNode] = []

    for child in LibXMLDOM.childrenOfNode(node) {
      switch LibXMLDOMGetElementType(child) {
      case .ELEMENT_NODE:
        children.append(LibXMLNodeParserDOM.createTreeRecursive(child))
      case .TEXT_NODE:
        text = LibXMLDOMGetText(child)
      default:
        break
      }
    }

    return XMLNode(name: name, text: text, children: children)
}

That’s pretty much it, there’s a little more in terms of plumbing and error handling, but its not too hard to take an XML Document from libxml2 and get the data into Swift.

Implementation 2: Swift Structure from libxml2 Buffered Reader

Looking at the first this means that the XML Document is built twice. Once to read the document into libxml2 and the second to extract out the Text Nodes and Element Names into a Tree of Swift data structures. The costs of allocating a bunch of Swift structures and objects could be significant so there will likely be a performance benefit to only building the tree once.

The Reader interface for libxml2 is an interface to the XML document that will incrementally read each component of the document in a depth-first fashion. The Current Node is repeatedly advanced by calling the xmlTextReaderRead function until the end of the file has been reached, or an Error has occurred in parsing. Using the Reader, a Swift Data structure can be built without having to create a DOM in libxml2. It is possible however, that the overhead associated by advancing through the Reader buffer could be greater than the cost of allocating the DOM in libxml2

A Lazy Sequence of nodes can be made for the Reader interface in the same way as the Tree interface. The Sequence is made from the more fundamental Generator:

internal typealias ReaderGenerator = GeneratorOf<(xmlTextReaderPtr, LibXMLReaderMode)>

private class func makeRecursiveGenerator(reader: xmlTextReaderPtr) -> ReaderGenerator {
    return GeneratorOf {
        if reader == nil {
            return nil
        }

        let result = LibXMLReaderMode.forceMake(xmlTextReaderRead(reader))
        switch result {
        case .CLOSED: return nil
        case .EOF: return nil
        case .ERROR: return nil
        case .INITIAL: return nil
        default: return (reader, result)
        }
    }
}

With the Reader approach, an error can occur at any time and therefore needs to be handled during the reading process. This is passed back in a tuple of both the pointer to the current node and the status of the read operation. It is then wrapped in a Sequence for the Higher-Order operations and the pointer must be advanced one time on the first read:

internal typealias ReaderSequence = SequenceOf<(xmlTextReaderPtr, LibXMLReaderMode)>
private class func wrapInSequence(reader: xmlTextReaderPtr) -> Result<ReaderSequence> {
    if (reader == nil) {
        return Result.Error(self.error("Could Not Parse Document, no root node"))
    }

    var generator = LibXMLReader.makeRecursiveGenerator(reader)
    if generator.next() == nil {
        xmlFreeTextReader(reader)
        return Result.Error(self.error("Could Not Parse Document, no first node"))
    }

    return Result.value(SequenceOf(generator))
}

In the LibXMLReader class, a Context class is used as a container to hide the details of the interface between Swift and libxml. This is all wrapped up in a structure⁵ containing a Reader pointer and a Sequence which will advance the Reader as it is consumed. It also has a dispose method for resource cleanup of the manually memory managed libxml structures. Note that there should only be one Context per reader to avoid multiple freeing of the Reader:

internal final class LibXMLReader {
    internal struct Context {
        let reader: xmlTextReaderPtr
        let sequence: ReaderSequence
        var hasFreed: Bool

        init (reader: xmlTextReaderPtr, sequence: ReaderSequence) {
            self.reader = reader
            self.sequence = sequence
            self.hasFreed = false
        }

        func dispose() {
            if !self.hasFreed {
                xmlFreeTextReader(self.reader)
            }
        }
}

The Tree can be built up using a Recursive function, passing through the ReaderSequence

private class func parseRecursive(reader: xmlTextReaderPtr, _ sequence: ReaderSequence) -> Result<XMLNode> {
    if LibXMLReaderType.forceMake(reader) != .ELEMENT {
        let realType = LibXMLReaderGetElementTypeString(LibXMLReaderType.forceMake(reader))
        return Result.Error(LibXMLReader.error("Recursive Node Parse Requires an ELEMENT, got \(realType)"))
    }

    var name = LibXMLReaderGetName(reader)
    var text: String?
    var children: [XMLNode] = []

    if LibXMLReaderIsEmpty(reader) {
        return Result.value(XMLNode(name: name, text: text, children: children))
    }

    for (reader, mode) in sequence {
        let type = LibXMLReaderType.forceMake(xmlTextReaderNodeType(reader))
        let typeString = LibXMLReaderGetElementTypeString(type)
        let modeString = LibXMLReaderGetReaderMode(mode)

        switch type {
            case .TEXT:
                text = LibXMLReaderGetText(reader);
            case .ELEMENT:
                let childResult = self.parseRecursive(reader, sequence)
                switch childResult {
                    case .Error(let error): return childResult
                    case .Value(let box): children.append(childResult.toOptional()!)
                }
            case .END_ELEMENT:
                assert(name == LibXMLReaderGetName(reader), "BEGIN AND END NOT MATCHED")
                return Result.value(XMLNode(name: name, text: text, children: children))
            default:
                break
        }
    }

    return Result.Error(LibXMLReader.error("I don't know how this became exhausted, unbalanced begin and end of document"))
}

There’s a little more in terms of book-keeping as the End and the Beginning of a nested element need to be matched up, but it should look very similar to the DOM based approach.

Implementation 3: Navigate the libxml2 DOM with Swift

The Building of the tree as a fully realized Swift data structure is a simple implementation and all the work is done up front, but it certainly isn’t the least resource intensive. In the previous implementation Strings for the Text and Element Name of an XML Element are created regardless of whether they are used or not. Even if the use of the Reader interface results in a faster implementation than the dual-tree interface of the first Implementation, redundancy in data that is read could mean a lot of wasted effort.

There’s no reason that we can’t just wrap a whole libxml2 Tree structure in a Class implementing XMLParsableType that knows how to fetch the Text and Children of an Element from a current Node Pointer:

public final class LibXMLNavigatingParserDOM: XMLNavigatingParserType, XMLParsableType {
    private let node: xmlNodePtr
    private let context: LibXMLDOM.Context?

    internal init (node: xmlNodePtr) {
        self.node = node
    }

    internal init (context: LibXMLDOM.Context) {
        self.context = context
        self.node = context.rootNode
    }

    deinit {
        self.context?.dispose()
    }

    public func parseChildren(childTag: String) -> [LibXMLNavigatingParserDOM] {
        let foundChildren = filter(LibXMLDOM.childrenOfNode(self.node)) { node in
            LibXMLDOMGetElementType(node) == LibXMLElementType.ELEMENT_NODE && LibXMLDOMElementNameEquals(node, childTag)
        }
        return foundChildren.map { LibXMLNavigatingParserDOM(node: $0) }
    }

    public func parseText() -> String? {
        let foundChild = findSeq(LibXMLDOM.childrenOfNode(self.node)) { node in
            return LibXMLDOMGetElementType(node) == LibXMLElementType.TEXT_NODE
        }
        return foundChild >>- LibXMLDOMGetText
    }
}

The LibXMLDOM.Context is just a internal class responsible for cleaning up the manual-memory-managed libxml2 DOM document when the root node is deallocated. Again, there is only one Context per Reader:

internal final class LibXMLDOM {
    struct Context {
        let document: xmlDocPtr
        let rootNode: xmlNodePtr

        init (document: xmlDocPtr, rootNode: xmlNodePtr){
            self.document = document
            self.rootNode = rootNode
        }

        func dispose() {
            if self.rootNode != nil {
                xmlFreeDoc(self.document)
            }
        }
    }
}

Still very simple and that’s about all that is required to implement XMLParsableType protocol. The creation of the Swift Strings for the Text of an Element are deferred until they are actually needed.

Implementation 4: Navigate with a Reader

In Implementation 2, a Reader was used to navigate the XML Document and extract all of the Elements and Text Nodes as a Swift Data structure. Instead of building a Swift data structure, it should be possible to extract out Text from Reader without needing a secondary data structure at all. This should work well in circumstances where the amount of data extracted from the whole document is minimal.

This will result in a substantially different implementation to the previous three and will be worth implementing at a later date. It is currently unimplemented in the XMLParsable project.

Testing

One of the great aspects of building an XML Parser in this way is that the implementations are trivial to swap out and can be tested against for correctness against the same Parsing Code. Additionally, we can use the new Performance Testing features of Xcode 6 to see how the performance varies in these differing implementations. The code is all available on GitHub for your viewing pleasure.

The correctness testing is relatively simple, just run the same suite of tests over a fixed XML resource and verify that all of the properties in the Model are set to the values in the resource.

Xcode 6 makes performance testing easy too. The same XML parse to Model decode task can be stuck inside a measurement block and repeated a number of times to eliminate any performance fluctuations of the test host. The Performance Tests should be designed in such a way that they can expose some the performance characteristics of each implementation with a different data set. One of these characteristics is that the amount of unused or redundant data in a document can massively impact performance.

The small suite of performance tests in XMLParsable test the same source data on each of the implementations on a number of axis, using NSData vs. using Files and a small XML file vs. an XML file with a lot of redundant data:

Some observations from the results:

• There isn’t a huge difference between NSData and reading from a file. This suggests that File I/O isn’t much of a bottleneck. There won’t be any noticeable difference between decoding XML from a File, or from NSData returned from a HTTP request.
• When a small XML document is used, there’s not much of a performance difference between all of the implementations. There may be some other bottlenecks worth identifying in the system, but the time taken to run 10 iterations is very small.
• The two implementations that build a Tree of Nodes in Swift have order-of-magnitude worse performance than the DOM-wrapper in the XML document with lots of redundant data. This highlights the differences in performance between building a pure Swift data structure and keeping the data in libxml itself. This doesn’t make Swift slow by any means, it just highlights the tradeoff of abstractions & safety against raw performance. Moving pointers around a region of purpose-allocated memory will always be faster than a runtime that checks types, reference counts and dereferences pointers across discretely allocated data structures.

Conclusion

The results prove that the performance characteristics of different implementations of an XML Parser can be exposed when using different data sets. The small gap in performance between implementations widens as the size of the document increases. This is likely due to the cost of object and structure allocations. Reducing allocations in libxml2 or Swift will squeezes out more performance.

I’d probably opt to use ‘Implementation 3’, which wraps the original libxml DOM in the XMLParsableType protocol. It is by far the simplest and has very good performance. Even when resources are at a premium it can be better to opt for the simpler solution, even if it means sacrificing a little performance. Maintainability and understandability are important qualities in code, so that the code can be revisited and revised by others and understandable for years to come. As ‘Implementation 3’ is the most performant and simplest to implement, hard to argue that it is the one to go for!

I hope you’ve enjoyed this series of posts, I’d love to hear your thoughts and comments! Have a look at the XMLParsable project and run the tests for yourself! I’m @insertjokehere on Twitter & lawrencelomax on GitHub.

---

In the previous post we built a decode function to parse data out from XML and into an Animal Model using Imperative techniques. This required some efforts in order to satisfy some of the robustness requirements from the first post.

In this post we’ll cover how we can use Functional Programming techniques on top of the language features of Swift to decode XML to an Animal model. This post assumes that you are comfortable with the Parameterized Types and Generics. Sample Code for is available on GitHub.

Thinking Functionally

I’m using the fantastic and lightweight Swiftz for go-to implementations¹ of many Higher-Order Functions. It also has a great implementation of the Result type from the previous post.

One important concept in Functional Programming is that functions can be thought of as First-Class types just like any other variable or constant in code. Objective-C has had blocks for some time now, allowing us to think in terms of passing functions around; assigning a block to a property of a Object. However this isn’t the same thing as First-Class functions. Swift allows Functions to be First-Class whilst providing equivalence in functions no matter how they are declared, a Closure in Swift is just another kind of function just like a method or Free Function. Blocks are no longer the best way of passing around a computation, we can consider the equivalence of functions declared with func in a Global or Class scope with local Closures:

let intToNumber:  Int -> NSNumber = NSNumber.numberWithInt
let intToNumber1: Int -> NSNumber = { NSNumber.numberWithInt($0) }

By now it should be second nature to think of the concatenation of two Strings using the + operator. As functions are types like any other, two functions can be joined together just like a String.

public func •<A, B, C>(f: B -> C, g: A -> B) -> A -> C    // The 'compose' operator from Swiftz

func prefixer(prefix: String)(string: String) -> String { // Equivalent to String -> String -> String
    return prefix + string
}
func postfixer(postfix: String)(bar: String) -> String { // Equivalent to String -> String -> String
    return string + postfix
}

let happyPrefix = prefixer("😃") // String -> String. The first String argument of 'prefixer' is applied.
let sadPostfix = postfixer("😞") // String -> String. The first String argument of 'postfixer' is applied.
let infixer = happyPrefix • sadPostfix // String -> String. Two functions are joined, the output of the first becomes the input of the second.
let string = infixer("Good Morning") // "😃 Good Morning 😞"

In this example, the Swiftz compose operator (•) is used in conjunction with Curried Functions. This might look a little crazy for now, but two important concepts are:

1. A specialized function can be created from a curried function, by applying some, but not all of the arguments.
2. New functions can be created locally by composing other functions with fancy operators.

Building new Parsers

With the idea of composing functions without having to declare one can be carried over to our problem of decoding XML to a Model; A specialized Parser function can be made for each of the values that need to be parsed out in the Model. In order to construct an Animal Model three parsers are required, with the values placed into the context of a Result for failure information:

let kindParser: XMLParsableType -> Result<String> = ...
let nameParser: XMLParsableType -> Result<String> = ...
let urlParser: XMLParsableType -> Result<String> = ...

Further, we can think of each of the properties of a decoded Animal as the application of the above functions to the source XMLParsableType from the XML Document.

let kind: Result<String> = kindParser(xml)
let name: Result<String> = nameParser(xml)
let url: Result<String> urlParser(xml)

We’ve allready seen that we can use Swift’s Optional Chaining in our parser to limit the number of occurences of handling failure. However, Result isn’t a blessed by the language with special syntax for chaining. It would be great to get the same behaviour for the Result type.

Turns out that Swiftz has the following function declared²:

public func >>-<A, B>(a: Result<A>, f: A -> Result<B>) -> Result<B>

It is called ‘Bind’ and it can be described in the following way:

“If the parameter a is a Value case of the result apply the function f returning the Result of f. If the parameter a is an Error, just return the original Result”

Sounds the same as Optional Chaining! In fact Optional Chaining can be defined in this way. It should be possible to construct a Result<String> for the kind value of an Animal this way:

let kind: Result<String> = xml.parseChildren("kind").first >>- { $0.parseText() } // Compiler Error!

Damn, looks like the types don’t match up since the kind value should be a Result<String> instead of a String?. Besides, this looks much worse with needing to use an inline closure to invert the parseText 0-arg instance method on xml into a function that takes an xml parameter and executes parseText. There has to be a better way of doing this…

XMLParser: A Helper

A new approach is to create a few Helper Functions that build on top of the basic functionality of XMLParsableType³ to extract a Single child of a given Element name, providing the value in the context of a Result type, and accept the XMLParsableType as a parameter of function that isn’t bound to an instance:

public final class XMLParser {
    public class func parseChild<X: XMLParsableType>(elementName: String)(xml: X) -> Result<X> 
    public class func parseText<X: XMLParsableType>(xml: X) -> Result<String>
}

This can then be used to extract out the kind value with our understanding of the Bind⁴ operator:

let kind: Result<String> = XMLParser.parseChild("kind")(xml: xml) >>- XMLParser.parseText

Now we’re talking! The XMLParser.parseText function is not invoked directly and is placed into a chain of operations to extract out text of an XML Element. These functions are being joined as if it were any other type of data.

The name property of the Animal model is nested a few XML Elements deep, but it can be extracted by chaining the Parsing functions in the same way:

let name: Result<String> = XMLParser.parseChild("nested_nonsense")(xml: xml) >>- XMLParser.parseChild("name") >>- XMLParser.parseText 

The XMLParser.parseChild("name") expression evaluates to a function of XMLParsableType -> Result<XMLParsableType> instead of just a Result value. We’re seeing that currying is being used to create a specialized Parser function for a specific XML element, chained together in a sequence of functions.

This is the heart of Function Composition, with Bind performing the behaviour of continuing on success, bailing out as soon as an Error occurs. The possibility of failure within any of the sequences of operations is an essential property of this Parser. Previously this was handled by Optional Chaining and if statements in the Imperative version. Using >>- there isn’t a branching statement in sight⁵.

Common Operations

There appeares to be some common chained functions appearing. These can be extracted out and into the XMLParser class⁶ and built using the same Higher-Order Functions and other functions and methods that have been defined for XMLParserType

extension XMLParser {
    public class func parseChildText<X: XMLParsableType>(elementName: String)(xml: X) -> Result<String> {
        let textParser: X -> Result<String> = promoteXmlError("Could not parse text for child \(elementName)") • { $0.parseText()}
        return self.parseChild(elementName)(xml: xml) >>- textParser
    }

    public class func parseChildRecursive<X: XMLParsableType>(elementNames: [String])(xml: X) -> Result<X> {
        return elementNames.reduce(Result.value(xml)) { (parsable: Result<X>, currentElementName) in
            return parsable >>- self.parseChild(currentElementName)
        }
    }

    public class func parseChildRecusiveText<X: XMLParsableType>(elementNames: [String])(xml: X) -> Result<String> {
        let textParser: X -> Result<String> = promoteXmlError("Could not parse text for child \(elementNames.last)") • { $0.parseText()}
        return self.parseChildRecursive(elementNames)(xml: xml) >>- textParser
    }
}

reduce is another Higher-Order function of the Sequence type making an appearance. Using it means that the parseChildRecursive doesn’t need to be implemented in a recursive manner.

These functions can now be used with in the Animal decode function to make extracting data from the XML more obvious:

let kind: Result<String> = XMLParser.parseChildText("kind")(xml: xml)
let name: Result<String> = XMLParser.parseChildTextRecursive(["nested_nonsense", "name"])

Lovely.

Parsers for Other Types

We’ve seen that the definitions inside the XMLParser Helper don’t provide methods for interpreting the String of a XML Text element as every possible Type as Numeric and Complex types within an XML document are represented in textual form. Our Model classes are concerned with Numeric types such as Double and Int, so the parsing of these types will need to be incorporated into the decode method.

Unlike JSON, there is no explicit syntax for a Numeric value, so the interpretation of these types isn’t a requirement of an XML Parser. Instead of bloating the XMLParser class with every variation of interpreting Text as a other Types, the Parser functions for values in the Model can be composed with XMLParser functions and other functions that interpret Strings as the other types.

Coercing a value from a String to a Numeric type may not always work. The String ‘14123’ can be interpreted as an Int but the value 134djk23 cannot. Again, this falls into our notion of Model decoding failure. The NSNumberFormatter class is a Cocoa way of interpreting Strings as Numbers, we can write an extension for the Int type to intepret a String as an Int, with Optional.None used to representing failure.

public extension Int {
    public static func parseString(string: String) -> Int? {
        let formatter = NSNumberFormatter()
        let number = formatter.numberFromString(string)
        return number?.integerValue
    }
}

As previously mentioned, Cocoa APIs in Swift expose failure as the nil/.None case of an Optional Type[^nserror-failure]. However, our Parser requires the additional information in a Result type. One approach is to extend Cocoa classes with additional methods that return a Result instead of an Optional, but this might not be the ideal solution⁷. Instead we can again think in terms of Function Composition to make a function that returns Result<Int> instead of Int?

For example, a definition of toiletCount requires interpreting a String in the XML as an Int in the model. This function can be built from closures:

let toiletCountParser: String -> Result<Int> = { promoteDecodeError("Could not parse 'disabled_parking")(value: Int.parseString($0)) }

Or we can use the Compose and Bind Operators again:

let toiletCountParser: String -> Result<Int> = promoteDecodeError("Could not parse 'disabled_parking") • Int.parseString
let toiletCount =  XMLParser.parseChildText(["facilities", "toilet"])(xml: xml) >>- toiletCountParser

The Animal and Zoo Models both implement the XMLDecoderType protocol. In this case there are a number of Animal Models belonging to a Zoo, so the decode functions can be reused to extract out each of the Animal Models contained in a parent Zoo:

let animals = XMLParser.parseChild("animals")(xml: xml) >>- XMLParser.parseChildren("animal") >>- resultMap(Animal.decode)

There is another function that takes the output of XMLParser.parseChildren as an input, of type Result<[X]>. The resultMap function is like a regular map on an Array, except with the return of a Result.Error if any of the applications of the map function fails:

public func resultMap<A, B>(map: A -> Result<B>)(array: [A]) -> Result<[B]>

Applicatives

Now we have everything we need to extract values out of XMLParsableType and into Result containers for each of the values that need to be extracted. Now the Result values need to be chained in the following way:

“If all of the Results corresponding to each of the Values are Successful, return our Model with the Values from the Result context applied to the Model Constructor function, otherwise just return the first Errored Result.”

Let’s turn back to Curried Functions, this time to a Curried ‘Constructor’⁸:

static func build(kind: String)(name: String)(url: NSURL) -> Animal {
    return self(kind: kind, name: name, url: url)
}

By adding a little more context to each of the applications of Higher-Order functions, we should be able to follow what is going on as the Animal.build function down to a Result<Animal>:

// These are previously defined using Higher-Order functions and XMLParser
let kind: Result<String> = ...
let name: Result<String> = ...
let url: Result<NSURL> = ...

// The Build Function is a Curried function that will yeild functions with every application until we have the Animal value.
let build: String -> String -> NSURL -> Animal = self.build
// Each of these stage gets the Animal structure closer to being initialized
let first: Result<String -> NSURL -> Animal> = self.build <^> kind
let second: Result<NSURL -> Animal> = first <*> name
let third: Result<Animal> = second <*> url

This feels a lot like solving a Mathmatical equation by reducing the variables to balance each side of the equation. There are a few more Higer-Order functions that are being used here, doing the heavy lifting of pulling values and functions out of a Result context executing a function and then sticking the return value back in a Result again.

Firstly, the fmap operator:

public func <^><A, B>(f: A -> B, a: Result<A>) -> Result<B>

It can be thought of in the following way:

“If the Result a is an Value apply the function f and place the result back in a Result, if the Result a is an Error, just return the Result immediately”

The property of this operator is that the applied function f doesn’t have to ever have to be aware that the applied argument has been part of a Result context.

Secondly, we can take a look the apply operator:

public func <*><A, B>(f: Result<A -> B>, a: Result<A>) -> Result<B>

It too can be described:

“If the Result f is an Error just return it immediately, if the Result a is an Error just return it immediately. Otherwise pull the function out of f and the value out of a, apply a to f then place the returned value in a Result”

Again, these functions themseves no longer need to be aware that values are contained in a Result context.

Between <^>, <*> and >>- all the combinations of applying Values to Functions inside and outside of a Result are covered. These Operators allow us to use pull values and functions into-and-out-of Result contexts, using functions that don’t have to be aware of the fact that the values and Functions may have come out of a Result⁹.

Teach the Controversy

There’s a lot to take in, some of the benefits should be clear, others are a bit more subtle. I don’t deny some of this doesn’t have a steep learning curve, it may go against many years of experience with languages that don’t have semantics for dealing with Functions as things that can be combined in these ways. All of these concepts and implementations are nothing new and are the products of high-shouldered giants. With the modernity of Swift we have the opportunity to incorporate more Functional Programming into Native iOS and Mac Developments. Given that there are certain programming techniques that can’t be carried over from Objective-C, its worth considering incorporating some Functional Programming if it fits the task at hand.

There’s a lot of concern that Operator Overloading is going to lead to a lot of Smart People doing some very silly things¹⁰ for the purpose of making dense, concise and far-too-clever code. The reality of these operators is that they are small in number and aren’t specific to just Optional and Result. As Arithmetic Operators deal with Numeric values, Functional Operators operators deal with the transformation of Functions themselves. These concepts are taken from other languages, so they aren’t at all specific to Swift.

There’s also some new terminologies and another way of describing the interaction of components, this isn’t less true of a Domain Specific Language that we can graft on top of Objective-C[^objc-dsls] or with a more flexible language. Sure, Objective-C has the benefit of a preprocessor to graft new features onto the language to and reduce the amount of boiler-plate required¹¹, but these can become pretty opaque over time.

However, unlike a Domain Specific Language, all of the code used is valid Swift, there is no transformation from a language in one domain, to a syntax that the compiler understands. The Compiler is fully aware of the Types of all of the transformed Functions leaving no room for ambiguity. The chances of a runtime error because of a keypath not existing or the type being different to the one expected by the code are greatly diminished.

I honestly believe its worth taking a jump at the conceptual hurdle, whether it is for writing code that is more robust and predictable or to satisfy a curiosity of learning something new.

Thanks for Reading! I suggest reading some other brilliant posts that do a better job than I of laying this all out! As a bonus, the next post will focus on how XMLParsableType can be implemented in a variety of ways.

---

In the previous post I talked about some of the possible requirements for a Parser that extracts data from a Serialization Format and places it in a Language-Native Model. In this post I’ll cover how an XML Parser for a Model Object can be built using some of the familiar Imperative features of Swift. Sample Code for this and future posts available on GitHub.

Model and XML

Let’s define a hypothetical XML that we wish to parse:

<zoo>
    <animals>
        <animal>
            <kind>cat</kind>
            <nested_nonsense>
                <name>grumpy</name>
            </nested_nonsense>
            <url>http://en.wikipedia.org/wiki/Grumpy_Cat</url>
        </animal>
        <animal>
            <kind>cat</kind>
            <nested_nonsense>
                <name>long</name>
            </nested_nonsense>
            <url>http://knowyourmeme.com/memes/longcat</url>
        </animal>
        <animal>
            <kind>dog</kind>
            <nested_nonsense>
                <name>i have no idea what i'm doing</name>
            </nested_nonsense>
            <url>http://knowyourmeme.com/memes/i-have-no-idea-what-im-doing</url>
        </animal>
    </animals>
    <facilities>
        <toilet>42</toilet>
        <disabled_parking>true</disabled_parking>
        <a_random_assortment_of_characters>sdfdf821n9sfa</a_random_assortment_of_characters>
        <seriously>
            <crazy_nested>
                <drainage>Good</drainage>
            </crazy_nested>
        </seriously>
    </facilities>
</zoo>

The Model includes the parts of the XML that our Application cares about and ignores others:

public struct Animal: XMLDecoderType {
    public let kind: String
    public let name: String
    public let url: NSURL
}

public struct Zoo {
    let toiletCount: Int
    let disabledParking: Bool
    let drainage: String
    let animals: [Animal]
}

This is simple and immutable, the Parser forms part of the backend for the User Interface to consume. There’s no reason for the User Interface to be able to manipulate these Models directly.

An Interface to XML

Stubbing a Protocol or Interface is a great way of getting to grips with the problem that needs to be solved. It also helps to determine what is necessary for to implement, as well as the details that can be ignored to solve the problem at hand. There is also a bunch of previously laid out requirements to be considered

In parsing this XML¹. I’ve made a few assumptions:

1. I care about the data contained in Text Nodes.
2. I need to be able to recursively address Elements within a Tree-like structure.
3. I need to be able to enumerate Elements of the same name at the same level in the tree.
4. I don’t care about anything else (namespaces, attributes, schemas).

With those assumptions in mind, a Protocol for defining how data can be extracted from a Parsable XML Node can be made:

public protocol XMLParsableType {
    func parseChildren(childTag: String) -> [Self]
    func parseText() -> String?
}

“That’s It?”. Yep. Everything else can be composed on top of this minimal protocol; more complex data extraction functions can be built on top of these fundamentals. It’s easy to define the responsibility of this Protocol in one sentence:

“An XML Parsable has an ordered collection of Child Parsables and may have an associated String of Text”

Protocols are permitted to have a recursive definition, using the Self placeholder type. How and where the underlying XML document is stored is left to the implementing class/struct/enum². As we will see in Part 4, obtaining a child may be implemented by traversing a fully reified data structure, or moving a cursor partially visible representation of the structure.

As well as a representation of the Data Serializations itself, there needs to be a consistent way of defining that a Model can extract out values of Data Serialization. The entry point can be defined in terms of a decode protocol³ that the Model structures should implement:

public protocol XMLDecoderType {
    class func decode<X: XMLParsableType>(xml: X) -> Result<Self>
}

This function will be where the action is and can be implemented in the Model type definitions themselves or separately as Extensions. As XMLParsableType has a Self requirement, the usage of an XMLParsableType protocol needs to be satisfied with Generics.

Surfacing Failure

In the interface for XMLParsableType, failure is indicated with the return of the None case of the Optional enum. An Optional surrounds the value with a Context describing the availability of the value. The absence of a value in a XMLParsableType indicates some kind of failure, but without any information about how the Error came about.

Some Cocoa APIs use nil as the return value represent failure⁴ alone, but idiomatic Cocoa will also include a by-reference NSError to return Error Information in the event of failure. Exceptions are being weeded out of Cocoa API for all but programmer error. NSError has the an associated “Error Domain” as well as String Description of the cause of the failure. This can be massively helpful as finding the code or resource that is responsible is a nightmare if the only information is “a problem occurred somewhere”. For example the NSJSONSerialization API will give the line-number of a syntax error in a JSON resource.

Moving to a Safe Swift world, return of an NSError and a possible value can be encapsulated in the same value using an Enumeration with Associated Values, rather dereferencing pointers. This is the Result with the same availability semantics as an Optional, with additional failure information provided with an NSError in the failure case:

public enum Result<V> {
    case Error(NSError)
    case Value(Box<V>)
}

As there are potentially many sources of failure in the decode method, it is handy to write a Helper Method that can “promote” an Optional to a Result with an Error if the Optional does not contain a value. This will populate the NSError with a default Error Domain and attach a User Defined message:

public func promoteDecodeError<V>(message: String)(value: V?) -> Result<V>

An Imperative Approach

From the previous post, I mentioned that the lack a dynamic runtime in Swift will mean that bringing over some Objective-C programming techniques will be impossible. As a Swift Parser won’t be able to use these techniques, a more traditional approach to parsing will have to be used⁵.

Failing-Fast was outlined as a potential feature of a Parser in the previous post, the implementation of an XML-to-Animal decoding function will need to take this into account. The following Imperative approach shows how it may be possible to extract a Result<Animal> from an XMLParsableType:

static func decode<X: XMLParsableType>(xml: X) -> Result<Animal> {
    if let kind = XMLParser.parseChildText("kind")(xml: xml).toOptional() {
        if let name = XMLParser.parseChildText("name")(xml: xml).toOptional() {
            if let urlString = XMLParser.parseChildText("url")(xml: xml).toOptional() {
                return Result.value(self(type: type, name: name, url: NSURL.URLWithString(urlString)) )
            }
            return Result.Error(decodeError("Could not parse 'urlString' as a String"))
        }
        return Result.Error(decodeError("Could not parse 'name' as a String"))
    }
    return Result.Error(decodeError("Could not parse 'kind' as a String"))
}

This doesn’t look great. The nesting is terrible, the Failure and Success conditions are separated around the conditional. In this case, there are only three values, a Model with more properties would make the nesting significantly worse. In Objective-C this can be better tackled by returning early on failure, however this would require lots of force unwrapping⁶. Conditional Statements are required as Failure when one of the values is missing is a requirement of our Model and is guaranteed to exist in the XML.

Despite these problems, there are some patterns are emerging. Firstly, it looks like extracting the Text from a Child is a common enough task that can be converted to a function in its own right. Obtaining a Child’s Child’s looks like an interesting area for some more meaningful functions. Optional Chaining is being used to avoid further conditionals.

Most importantly is that the nesting is an Imperative way of implementing that the successful condition for this function is dependent on the availability of three values in the XML. Moving our understanding of the pattern from an Imperative model to a Declarative model will be crucial to building a better way of constructing a decode function.

Next Time…

Next time, we’ll take a Functional approach to the decode method, allowing us to think at a much higher level about how a Model is built.

This the first in a series of four posts on Parsing code in Swift. This first post takes a look at the development landscape for Cocoa, Swift, Objective-C & Parsers. Some of the stuff covered here is also handled very well by Rob Napier and Tony DiPasquale so make sure that you check out their posts if you haven’t already.

Introduction

iOS Development has been a fun ride over the last few years. Frameworks are added and improved and generally speaking follow some common idioms. UIKit is well-trodden ground with significant changes as the level of interactivity and responsiveness of what a mobile experience is moves ever forward. When new API introduced or refined, compatibility issues occur, we fix them, we add delete some code that new API subsumes, we write some new features using the new API.

The Backend of an Application is still a bit of a frontier. With no BackEndKit to provide a unified interface of a Network Connections and the Persistent State of the Application, there are no ‘rails’ to start on and how a Backend is built can vary enormously from Application-to-Application. A expanding world of Libraries has sprung up to simplify and consolidate the thinking of how back-end components intact in Objective-C. Much of the dynamism and flexibility of the Objective-C runtime as well as preprocessor macros is leveraged in order to kill repetitious code with metaprogramming. Now that a new Programming language has arrived and new OS features that all but a modular design the time is now to look at best practice in iOS App Backends.

There is a lot of concern how Swift can (and can’t be used) used with existing ways of getting-stuff-done in Objective-C. WWDC 2014 and the following months have had a ton to learn, so lets review a little of what we know about how working with Swift will effect iOS Development.

Cocoa & Swift

Swift Interoperability with Cocoa is a mostly known quantity. Objective-C interoperability is deeply ingrained in the language, regardless of the philosophical departures that Swift makes from Objective-C. Cocoa will remain, repeated casting will be inconvenient, Strong Types with the Optional type contract will allow us to reason better about the state of the Application. Smaller components of the Application can be migrated to a Swift world in a piecemeal way without drastically changing the underlying architecture due to the extent of the interoperability features.

There’s a lot of new language features, with a focus on moving up the levels of abstraction¹. Having Lazy Sequences as part of the Standard Library and setter side-effects as part of the language all show that things are getting a lot more modern.

The compiler is now doing a lot more with type inference can be used to increase code density without sacrificing readability or intent. Functions are types like any other value, people are rejoicing as the syntax of the language can be pared back by removing the need to worry about pointers.

Headers & Classes

Having headers is fantastic from a documentation point of view, but the evils of duplication makes can make them unwieldy as well as informative. Swift leans on Header generation² and access control to automate the process of generating an Interface from the Implementation itself. Implementation details don’t need to be leaked and API stubs don’t need to be copy-pasted between .h and .m

The implementation of both State and Behaviour become considerably ‘cheaper’ to write, re-write and update. Classes require a fraction of the number of characters and lines-of-code to achieve the same results in terms of defining Properties and Methods to their Objective-C brethren³. By making the time taken to write and re-write a Class significantly easier, a great deal of the resistance of decomposing Classes into smaller and more testable Units is removed. The Units become smaller and easier to test, revising code becomes easier as part of a virtuous circle.

Metaprogramming & Dynamism

Chief among the concerns with Swift is how we will cope without the dynamism that Objective-C offers. The malleability of the runtime allows for the creation and injection of functions at runtime⁴. Property Mapping libraries and Model Descriptions inspect classes at runtime, defining the interaction of Classes at a much higher level of abstraction. Reducing boiler plate code isn’t just about convenience at the current moment in time, it reduces the entropy of the codebase over time.

However, applying these methods will just not work in Swift. Pure Swift constructs are closed at compile time and cannot be tinkered with at runtime. There is no preprocessor available for code generation before it hits the compile. Safety is king, requiring the code to be totally unambiguous about the semantics of the code if there is insufficient information for the Compiler to reason with. There is no Reflection package or framework for performing the same kind of interactions with Classes at runtime, so this seems like a huge problem.

However there are other ways of constructing and building Applications, using Abstractions that seem far away from the Gang-of-Four, Object Oriented Programming and the more dynamic idioms of Objective-C. Functional Programming provides us with abstractions that help us to dispense with the repetition of Imperative programming but without sacrificing safety and predictability. These techniques can be employed without the need to read 42 tutorials about what a Monad is and isn’t. Functional Programming has a few fundamental particles, once we learn how they can be applied, the mathematics behind them become a mainly academic exercise that can be ignored if we want to. I’ll cover this in the third article in this series.

Parser Requirements

Let’s take a look at the how we can go about one component within an Application in Swift, Parsing.

I’m defining ‘Parsing’ to be the process of extracting data from Data Serialization format such as XML or JSON, to a native Model structure, valid and usable by the rest of the Application. The Parser itself can have some or all of the following requirements:

1. Check that a Value exists for a given Key at the current level, or navigate the Data to a deeper level.
2. Distinguish between a terminal value in the Data Serialization itself versus the value not existing at all. A null value in a JSON needs to be represented in a distinct way to a value not existing at all.
3. Coerced a value in the Serialization format into the type of the value that is expected in the Model.
4. Fail early if the the Serialization Format differs in any way from our expectations, without crashing.
5. If Parsing failure occurs, five valuable output to make it easier to isolate whether the Parsing code’s assumptions about the structure of the data are wrong, or that the structure of the data itself is wrong, or both a wrong.
6. Provide defaults, or ways of deterministically deriving defaults for values that are known to be optional in the data structure.
7. The interface to the Serialization format’s contents should provide a level of abstraction consistent with how it should be used.
8. Don’t let any external state effect the Parsing unless absolutely necessary. Really, don’t.
9. Seriously, don’t even think about modifying external state. Being able to make a parser concurrent with the rest of the application makes everybody happy.
10. It should be easy to Parse serialized data from a variety of sources into the parser. The data may come bundled with the Application, from a HTTP request, or a fixture in a Unit Test.
11. Write Unit Tests for all of the boundary conditions, so we can be super-certain that our understanding of the serialization format holds true in the implementation.
12. If our Unit Tests don’t cover all of the possible conditions in the structure of the format and a new condition is discovered, write a test for to ensure as much of the information about the structure of the data has been captured as possible.
13. Don’t make the interface for pulling data out of the Serialization more expansive than necessary. If functionality is required that can be expressed in terms of more fundamental extraction functions, define it as a composition of the more fundamental functions.

This is not an exhaustive list of requirements and it has already exhausted you and I. Admittedly, some of these requirements are true of Software Development generally, but it is always worth thinking about.

In Objective-C, Parsing can be a minefield of typing and branching. I don’t think I’d be going out on a limb to say that it is one of the most fragile parts of most codebases. Because of its fragility, if an Objective-C Application has any Unit Tests whatsoever, the Parsing components will have the best coverage. This is a big driving force behind Libraries such as the brilliant Mantle

Regardless of the awesomeness of an Objective-C library for parsing data, it won’t ever be possible to inspect the expected type of the destination property in the Model at runtime and thereby avoid issues with Model Objects having values of the wrong type assigned⁵. Not only that but Objective-C doesn’t fail-fast if an Object assigned an Object of a class differs from the expected class that the assignee Object expects⁶. This leads to explosions that occurs far away from the original erroneous assignment⁷.

However, it would appear that the situation is even worse for Swift. Without much in the way of metaprogramming or some runtime magic it looks like the process of parsing will be incredibly repetitive and laborious.

XML

XML is an interesting serialization format because it isn’t JSON. It’s a format that we like to chuckle about and deride because of the complexity, but the reality is there are many APIs that Applications need to connect to that use XML. I think that many of the problems that we have with parsing XML comes from the fact that the abstractions we use to extract data from XML aren’t that great. Handling all of the failure points in Parsing is a hard task and that’s why we want to automate it with metaprogramming and make it bulletproof with Unit Tests.

XML is also format because there is a good amount of variety in the ways that we model an XML document and its content in API⁸. Typically JSON is just extracted into the native Array, Dictionary, String & Numeric types of the targeted programming language. XML grew in popularity at a time when system resources like available memory were at much more of a premium than they are today. Constraints like this tend to drive innovation in a few directions, which make the process of parsing a little bit more interesting that pulling values out of a language-native Dictionary object.

If you aren’t a crazy person like me you’ll probably stick a Webservice in front of whatever else you need to consume and do all the complex data processing on a Server⁹, then spit everything out in a RESTful JSON service that normalises any intricacies and madness in the original data. However the reality isn’t always so ideal and its up to the Client Software to process data from a variety of Sources and Serialization Formats.

Next Time…

In the next post, I’ll be a crazy person and create a simple XML interface in Swift, and extract data out of it in an Imperative style.

Note: I wrote half of this in a crappy coffee shop the weekend before WWDC 2013. There have since been some great blogposts about iOS, Unit Testing & TDD. I hope you get some benefit out of this post. I’ve split it into two parts lest anyone fall asleep

A Bad Start

I can clearly recall my initial, visceral reaction to Unit Testing when learning JUnit at the end of a ComSci 101 module at University. It was one of confusion at the seeming pointlessness of it all. This is a common view among Unit Test opponents; ‘Write good code, and you won’t break anything!’. Despite everything that books, lecturers and my peers were saying, I continued to dismiss the efficacy of Unit Testing entirely.

I had these (admittedly lazy) opinions validated by Agency work, where the continual churn of fixed-price projects, out severely limits the amount of time to get anything done. In hindsight I was making excuses for lacking the will to learn, practice or acquire the knowledge to write effective tests.

Additionally, Unit Testing doesn’t exactly have the greatest pedigree in the Objective-C development community. Many of the community’s biggest names are outwardly hostile to it.

A New Hope

In the world of Unit Testing there has been a lot of change recently; both in terms of my own personal views and of the tools available.

My views started to change when working on longer term and continuously developed products. As before, the Success of a Product is measured in terms of how it performs in the market. However, the quality of the codebase in terms of stability and documentation becomes increasingly important.

The excuses and laziness that prevented me from getting stuck into Unit Testing began to disappear when I had to spend less time is taken up with heads in a debugger. Hesitancy to make positive changes to the quality of your codebase through refactoring fades away when Unit Tests give you a safety net and sanity check to explore better ideas.

The quality and availability of tools has changed a great deal recently too. There are a whole host of Open Source libraries and frameworks that in practice can be an order of magnitude more expressive than the default OCUnit and XCTest. Apple are now fully on-board with Unit Testing, and to a greater extent, Continuous Integration in XCode 5

There are host of tools that take different approaches to how to go about the process of testing. The are monolithic frameworks such as Kiwi, or you can take an a-la-carte approach in libraries that handle specific tasks, such as specta for assertions, OCHamcrest for matching, expecta for asynchronous testing and OCMock for mocking Objects. With the help of CocoaPods, these frameworks & libraries are simple to integrate.

Dodging Dogma

At first, it may seem totally backwards, to make code changes just for the purposes of box ticking. This perception isn’t helped by the Cult of Unit Testing, which overemphasises the importance of Unit Testing to the point of confusing the means with the end.

Good iOS & Mac developers rightly care about better applications for End Users. This doesn’t exactly fit with with usual objective metrics and beyond programmer intuition, there are also ways of measuring how well have tested our code.

It may be necessary to take actions to fit Unit Testing into the way that you think and work. Some sections of an Application’s architecture have particularly big gains in terms of quality and yield great results in the future:

Test Cases as Documentation

There is a whole debate between BDD and TDD, which can be heaps of fun if you like to have unimportant arguments about methodoligies that achieve a similar result. Whatever approach you use, you’ll find that thinking about what your tests need to cover, crystallises behaviour as you write.

As individuals, it is all too common to over-estimate obviousness in behaviour, for the code we write. Ask a handful of engineers to solve a particular problem and you will receive a chorus of solutions. Header documentation is one way of expressing the behaviour of methods and classes, but this is behaviour that you wish to make public. Headers should remain succinct and avoid expressing internal, private behaviours. Unit Tests can certainly help out here.

When developing in teams of more than a few people, a great deal of programmer intent can get lost in a stream of commits. This is especially true of teams that operate remotely. Unit tests can be wonderfully expressive, self-documenting and clearly define the implementation. Commits containing behaviour with corresponding tests, gives explaination in a clear and often concise manner.

Unfortunately, even the best intentions have to be put aside sometimes and ‘compromises’ (hacks), may have to be used when in a tight timeframe. Documentation of these moments of voodoo is crucial in a team, this is often done through code comments. Some refactorings later the voodoo may be removed when a better implementation comes along. The voodoo is removed with out-of-date commit comments remaining as a source of incorrect documentation. Removing and amending tests during refactoring keeps consistency between the documentation and the implementation as well as a living commentary of code changes such as refactoring.

State trips you up

If you have a lot of internal state within one class, Unit Testing can get a bit messy. An Object with a large amount of accumulated internal state can require disabling a large number of methods with stubbing, or setting many private and public properties to get the Object in a state suitable for performing certain behaviours. This effort is mostly unnecessary, as there is often an implementation of a Class that can have the same behaviour without the same amount of state.

Not only can state pose a problem in terms of effort required to just run some simple Unit Tests, it can have negative effects on the predictability of an Application. Conditionals can be dependent on internal state. This makes modifying a Class for others a difficult task; often requiring intimate knowledge of how all the state interacts.

Mutable Global State can also be troublesome to the predictability of Applications. Classes may make assumptions about global state that are not true in the clean room of a Unit Test environment. Affecting mutable global state in one test case may causes false failure, or false success in successive tests. The correctness of tests is crucial to the future of your application.

One example of this is seen with abuses in the usage of Singletons. Unit Testing a singleton can be very troublesome, as Singletons can tend to contain all many assumptions about the data that they consume, as well as storing state at the instance level, which is in effect global. Increasing the data dependencies of a Class increases its fragility to changes in the rest of the Application.

At first it may seem that the solution to giant this is refactoring via chains of Singletons with more specific responsibilities. On face value this appears to be a solution, since state is siloed into separate classes, decreasing the amount of accumulated state per class. The assumptions about consumed data still exist, just over more Classes. Writing methods for these classes that take arguments and do not mutate internal state as far as possible makes Singletons far more testable.

Rather than avoiding writing tests because they are painful, there is an opportunity to consider whether an implementation needs to have so much of a reliance on instance and global state. Unit Tests can highlight these giant, messy machines, more effectively than skim reading an implementation. As has been wisely observed, ‘persistent state is the enemy of unit testing’

Dealloc is still a thing

Yes, the -dealloc method still exists in the wonderful land of ARC. ARC removes a lot of the boiler plate for object destruction, however destruction code may still have to be written; KVO must be cancelled, NSNotification observers must be de-registered and any C malloc() must be free()’d.

RSpec has been an inspiration for a few iOS Frameworks. It has a syntax that allows you declare setup and teardown actions in nested code blocks, shared between test cases. In these code blocks a fresh instance can be instantiated for each test case, where application code may be only ever use a single instance. By using a fresh instance for each test case, the cases do not make any false assumptions about the internal state of an object.

With the repeated creation and destruction of objects, problems from incorrect destruction code will rear their ugly head. Ignoring these problems in the present does not solve the problem, it only delays it. Unit Tests can act as stress-test for object lifecycle, profoundly affecting the ease of moving away from Singletons.

Side Effects made Explicit

In an imperative, object-oriented language such as Objective-C, setters may have behaviours beyond just setting a value in memory. We don’t write code in a world that enforces a strict mapping of input to output via a function, with no side effects. The idea of setter side effects may be distasteful and dangerous for some, since the public interface does not always convey this information. However, setter side effects can be used to encapsulate logic inside the receiving object’s class, rather than requiring the caller to call additional methods after a setter.

Test Cases can describe these side effects. For example, a setter may perform an asynchronous task that writes to file. Unit Tests make these behaviours explicit to other developers, as well as asserting that these behaviours should not disappear with future code changes.

Errors, Fails, Empty, Null

Even the most experienced engineer can omit error conditions and fail to program defensively. Being somewhat somewhat spoiled by Objective-C, it is easy to get a false sense of security for null checking as it isn’t a language that shits the bed anytime a method is called on null

It is wishful thinking that an implementation will stay frozen in time after a public release, since so many applications rely on Web APIs that can changed or be removed. Areas of responsibility that don’t handle certain failure conditions can blow up later when APIs change. //A collection will not be happy with nil keys or values and then the Application will be at the mercy of an exception handler.

Network requests can fail and return errors. A lazy stream of values can send errors, saving a file to disk can error and saving values in a database can fail. All of this requires some degree of defensive programming, adding to the size of a code base, at the same time as increasing its tolerance to changes in external APIs.

In Mobile we should strive for the leanest and most polished solution for the User. A lean Application is defined by its willingness to cut down on useless features and visual clutter, not minimalism in the number of lines of code. The process of writing Test Cases presents another opportunity to think about behaviours that we expect, such as operations on null objects and Errors being returned from an external service. This may help us program more defensively, if we write tests for these behaviours, crucial to the stability of our code in the future.

Wrap-Up

Unit Testing is traditionally seen as a way to validate the correctness of an implementation and ensure that it does not regress in the future.

You may have to make changes to your Application code to make it more testable. This may appear to be backwards at first, but it is worth persevering! Altering your implementation to make it more testable has benefits beyond correctness in the present, and preventing regression in the future. I hope to show you more of these effects in Part 2.

I recently saw this comment in a Hackernews Thread, in reference to the battery performance characteristics of major mobile devices. Emphasis mine:

Absolutely that’s a problem with Android. It’s a natural consequence of having a platform where apps are free to access pretty much any hardware in the background with no review (that’s a good thing if you’re a developer though).

However, Android gives you tools to see which apps are using the most battery, and you can uninstall them if they’re a problem.

At first glance, this makes sense. When a developer is given more freedom in the Operating System that they develop for, they have the freedom to make Applications for users. The argument that ‘open is better’ often boils down to this.

The Hacker Mentality of Open Systems has lead to so many great products such as the Raspberry Pi , Linux (and therefore Android) and tools that developers use the world over. The libertarian philosophy of doing as you see fit with the hardware that you own, is certainly romantic in the way it provides a romantic liberation from our closed overloads.

Unfortunately for the Hacker Mentality, nobody cares.

Or should I should say, ‘only a very small and vocal minority of people care’. If you are writing an App for non-technical users (and you most likely are), these users care about a few things:

1. Battery Life
2. Lots of fresh and up to date content
3. Availability on my Operating System
4. Not asking for creepy personal details
5. An attractive and friendly User Interface
6. An accessible User Interface
7. An interface that responds to my type of Device and my Touches
8. No confusing messages
9. Battery Life
10. Not jumping through elaborate hoops to get to content
11. Not having to continually maintain an App’s storage and memory usage
12. Battery Life

It is good to have a debate about which platforms are better for the User across a whole range of criteria. We can have conversations about which criteria are the most important. To see managing the memory usage of an Application in your OS as a feature, is to fundamentally misunderstand what Users really care about.

We now live in a world where a significant proportion of users have never seen what I know to be a save icon. Even more profound is users who have never had the pleasure of using a Task Manager before. An OS having a Task Manager is not a feature or a necessity anymore, but having decent battery life certainly is. Achieving better battery life without a Task Manager is the sweet spot.

If you see a task manager… they blew it

Implied in the statement ‘that’s a good thing if you’re a developer though’ is that what is ‘good’ for the Developer is good for the User. I’d argue the reverse is the case; what’s good for the User is good for the Developer, because when a User has a better experience, the Developer is likely rewarded.

This isn’t to discount the importance of those with products that are aimed at the Technically Minded. There are hundreds of thousands of users of Hacker News, Slashdot, Reddit and the like. But the source licence used in major components of an OS is not a criteria that the overwhelming majority of people really care about.

Those with techie tendencies should not be so arrogant to assume that those outside their circles, should care about the same things that they do. Caring about the value, simplicity, joy and beauty that Mobile Applications can provide to the masses is essential to being an awesome Mobile Developer.