Explore the use of fiddle to save ASCII plists using Xcode frameworks

[kylef]

We should be able to load DVTFoundation.framework dynamically using fiddle in Ruby which would allow us to save ASCII plists using the private API if available and Xcode is installed.

This provides a couple of benefits:

• No need to compile and install xcproj, this would be the standard behaviour of Xcodeproj.
• Removes the need to re-compile xcproj when Xcode changes.

Certainly not for the upcoming release. But I think in theory this could work and it would be great it we can get it to work.

/cc @alloy @0xced

[fabiopelosin]

🔥

[AliSoftware]

👍 hell yeah it would definitely be Xmas if we could do this 🙏

[alloy]

@0xced Do you think this would be feasible without any objc method calling semantics? i.e. C function calling only?

[segiddins]

@alloy in theory that's always possible by using the C objc runtime functions!

[alloy]

@segiddins Yeah definitely, but I guess I was wondering if it would be huge hassle, but I realise we could use the magic of Ruby to refactor some of that away.

[0xced]

After 15 minutes writing a thorough explanation of why xcproj requires to be bound to the Xcode version that compiled it, I found a way to get xcproj to work with any version of Xcode. So thanks for filing this issue. 😆 The result is at 0xced/xcproj@4e26298. I’m still waiting for feedback in #196 to release a new version of xcproj by the way. Of course, this doesn’t solve the integrated behaviour inside Xcodeproj problem but it’s still nicer that the current situation.

Now, to the actual issue of Xcode frameworks + fiddle to produce ASCII plist.

Producing the ASCII plist from an NSDictionary is straightforward once you’ve loaded the DevToolsCore.framework (not DVTFoundation.framework). It’s a matter of calling the -plistDescriptionUTF8Data category method on NSDictionary. It returns a NSData instance which is the ASCII (open step) representation of the plist.

Here is the bare minimum you’ll have to do:

@interface NSDictionary (PBXPListASCIIDescription)
- (NSData *)plistDescriptionUTF8Data;
@end

void LoadDevToolsCore(NSBundle *xcodeBundle)
{
    NSString *otherFrameworksPath = [[[xcodeBundle privateFrameworksPath] stringByDeletingLastPathComponent] stringByAppendingPathComponent:@"OtherFrameworks"];
    NSString *devToolsCorePath = [otherFrameworksPath stringByAppendingPathComponent:@"DevToolsCore.framework"];
    NSBundle *devToolsCoreBundle = [NSBundle bundleWithPath:devToolsCorePath];
    [devToolsCoreBundle loadAndReturnError:NULL];
}

int main(int argc, char *argv[])
{
    @autoreleasepool
    {
        LoadDevToolsCore([NSBundle bundleWithPath:@"/Applications/Xcode.app"]);
        NSDictionary *pbxprojPlist = …;
        NSLog(@"%@", [[NSString alloc] initWithData:[pbxprojPlist plistDescriptionUTF8Data] encoding:NSUTF8StringEncoding]);
    }
}

Note that there is no error checking at all and the Xcode path is hardcoded. A production solution should be much more robust! See my solution in xcproj for loading the DevToolsCore framework.

That’s just the Objectice-C starting point. Translating it to fiddle is going to be pretty ugly, something like this probably:

objc_msgSend(pbxprojPlist, sel_registerName("plistDescriptionUTF8Data"))

[alloy]

This should get its own write method, e.g. PlistHelper::write_xcode_project so that it can:

• Fallback to PlistHelper::write when it can’t locate the Xcode frameworks.
• Fallback to PlistHelper::write when it can’t load the Xcode frameworks (rescue exceptions).

[lilyball]

Loading DVTFoundation.framework (or DevToolsCore.framework) is a horrible idea. Truly awful. Dynamically loading and using private Xcode frameworks would be a complete showstopper for me and for lots of people I know, and would cause us to immediately delete CocoaPods from our computers and tell everyone we know to do the same.

Really, you just need to implement your own plist serialization routines. The OpenStep Plist format is pretty darn trivial to write out. The only complication is reproducing all the comments that Xcode puts in there.

Which brings us to the next argument against loading the framework: It actually doesn't output what Xcode does. Take an Xcode project file and run it through the above code, and you get output that has some comments, but doesn't have all the comments it should.

Notably, the above code only adds the // !$*UTF8*$! header and groups all the entries in the top-level objects key based on their isa value. Beyond that, every object is printed like a normal dictionary. However, Xcode actually treats most of the object types specially, adding comments that describe the name or location of the file, and in some cases even printing the dictionary on one line.

It should certainly be possible to document all of the rules that Xcode is observed using and reproduce them. An alternative approach is instead to create a DOM model that represents the contents of a project file and maintains all existing comments, and all ordering of values in the dictionaries. If done write, this DOM can be re-serialized and produce a byte-for-byte identical output. You could then modify the nodes in the DOM as needed. The downside to this approach is any new nodes that CocoaPods adds won't have the appropriate comments or formatting, which means Xcode would presumably add in the comments/formatting the next time it saves the project file. However, any existing nodes would be preserved, and as long as Cocoapods isn't renaming the files or changing the groups they live in, the comments should not bit rot (besides Xcode, would rewrite any that do the next time it saves the project).

This approach is obviously more work, but it allows Xcode to maintain control of the formatting of the project file, and is compatible with any future changes Xcode might want to make. The downside is that creating new nodes, and new project files (such as creating Pods.xcodeproj for the first time) will not be able to reproduce the correct formatting. The upside is that this approach should be less prone to creating conflicts; if all of Xcode's formatting is preserved as-is, then conflicts in that formatting cannot arise. Whereas if Cocoapods attempts to reproduce Xcode's formatting from scratch, the slightest deviation from Xcode's behavior can cause conflicts.

---

All that said, I think something has to be done here. The other day I spent several hours manually resolving conflicts in an Xcode project file from a long-lived branch, where both the branch and mainline had modified the project file and run pod update/pod install many times. Dealing with that was really quite horrible.

Also to note, I found a ton of UUIDs with values (null) in the project file, on both branches. The Xcode comments all marked them as being /* (null) in Resources */, and at least some of them had their UUIDs show up in a Resources build phase files list as well. But they didn't actually have any associated files, and deleting them all resulted in no ill effects. So somehow Cocoapods managed to create all these things, possibly through merges, possibly not. I have no idea. Either way, having Cocoapods handle project files much more gracefully would probably have kept this from happening in the first place.

[alloy]

Loading DVTFoundation.framework (or DevToolsCore.framework) is a horrible idea. Truly awful. Dynamically loading and using private Xcode frameworks would be a complete showstopper for me and for lots of people I know, and would cause us to immediately delete CocoaPods from our computers and tell everyone we know to do the same.

Please also provide actual arguments. Is this a principle issue? Is it a stability issue? (Because the idea is that code reverts back to using the public CFPropertyList API.)

Really, you just need to implement your own plist serialization routines. The OpenStep Plist format is pretty darn trivial to write out. The only complication is reproducing all the comments that Xcode puts in there.

You should assume that we are well aware of the costs and benefits analysis and that ‘the only complication’ is in fact the only reason why we’re “exploring” this approach, as the title states.

Which brings us to the next argument against loading the framework: It actually doesn't output what Xcode does. Take an Xcode project file and run it through the above code, and you get output that has some comments, but doesn't have all the comments it should.

Notably, the above code only adds the // !$*UTF8*$! header and groups all the entries in the top-level objects key based on their isa value. Beyond that, every object is printed like a normal dictionary. However, Xcode actually treats most of the object types specially, adding comments that describe the name or location of the file, and in some cases even printing the dictionary on one line.

I don’t know which code exactly you are referring to, but the PR code doesn’t work like you state at all:

$ irb -I lib/xcodeproj -r plist_helper
irb(main):001:0> path = File.expand_path('spec/fixtures/Sample Project/Cocoa Application.xcodeproj/project.pbxproj')
=> "/Users/eloy/Code/CocoaPods/Xcodeproj/spec/fixtures/Sample Project/Cocoa Application.xcodeproj/project.pbxproj"
irb(main):002:0> plist = Xcodeproj.read_plist(path); nil
=> nil
irb(main):003:0> p plist.class
Hash
=> Hash
irb(main):004:0> Xcodeproj.write_plist(plist, path)
=> nil
irb(main):005:0> exit
$ git status
On branch pbxproject-write-💥
Your branch is up-to-date with 'origin/pbxproject-write-💥'.

nothing to commit, working directory clean

As you can see, git does not notice any changes in the project file. Opening with Xcode, making a change, and undo-ing shows the same result:

$ open -a Xcode 'spec/fixtures/Sample Project/Cocoa Application.xcodeproj'
$ git status
On branch pbxproject-write-💥
Your branch is up-to-date with 'origin/pbxproject-write-💥'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   spec/fixtures/Sample Project/Cocoa Application.xcodeproj/xcshareddata/xcschemes/Cocoa Application.xcscheme

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    spec/fixtures/Sample Project/Cocoa Application.xcodeproj/project.xcworkspace/xcshareddata/
    spec/fixtures/Sample Project/DerivedData/

no changes added to commit (use "git add" and/or "git commit -a")

Here’s the actual fixture file so you can see for yourself that it does in fact contain all the comments.

It should certainly be possible to document all of the rules that Xcode is observed using and reproduce them. An alternative approach is instead to create a DOM model …

Indeed, there are many possibilities. I’d love to see work being put into what you think is the best possible solution and being shared with us. We prefer to discuss actual implementations, however raw they might be.

All that said, I think something has to be done here. The other day I spent several hours manually resolving conflicts in an Xcode project file from a long-lived branch, where both the branch and mainline had modified the project file and run pod update/pod install many times. Dealing with that was really quite horrible.

I’m sorry to hear that you had to spent so much time on that. Clearly this is not as it should be.

Please elaborate on your details if I got this wrong, but I read this as being an issue with your app Xcode project file being hard to merge?

In that case, it might have something to do with 0.34.x introducing a regression that made it save the user's project on every pod install / pod update when it had no business doing that. We’ve been fixing this and issues tangential to it.

To summarise, CocoaPods should never touch the user’s Xcode project, unless integration is necessary. i.e. when:

• you add a new target to the Podfile
• a CocoaPods update requires new settings

Also to note, I found a ton of UUIDs with values (null) in the project file, on both branches.

That does sounds really bad. Can you please open a new ticket that shows steps to reproduce this? I’m guessing it might be related to assigning new xcconfig files. (Although those shouldn’t be considered ‘resources’, but that might just be a red-herring.)

[alloy]

@kballard Bump?

[lilyball]

@alloy Sorry for the delay.

Please also provide actual arguments. Is this a principle issue? Is it a stability issue? (Because the idea is that code reverts back to using the public CFPropertyList API.)

If the behavior of the API changes, but the symbol still exists, then you won't know to revert to CFPropertyList and will misbehave (or crash). Similarly, this is a private API, not a public one, so it may not be resilient against unexpected input, or may have various caveats about its usage that you can't know via reverse-engineering.

Apps on OS X have had a long history of (ab)using private frameworks. That's tolerable (though still a rather bad idea) when the private frameworks ship on the system, because system updates are infrequent and frameworks only change to fix bugs in between major releases. However, what's being proposed here is using a private framework that ships with Xcode, which means it isn't tied to the OS. It can change radically with every single Xcode update if it wants to. Which is to say, using a private Xcode framework is orders of magnitude worse than using private system frameworks.

I don’t know which code exactly you are referring to, but the PR code doesn’t work like you state at all:

I was referring to the sample code pasted in this comment.

Glancing through the PR, it loads a handful of frameworks. I assume this is to get the objects that tell the plist formatter about the comments. This approach is not future-proof, because any objects that are added that exist in other frameworks will not be loaded. Furthermore, I'm not even convinced it works as-is today, because it's loading the Xcode3Core.ideplugin, and I assume that's because a number of these objects are defined in that plugin. Given that, it's possible there's another plugin that defines some object that simply hasn't been present in any xcode project that this code has been tested against.

I’d love to see work being put into what you think is the best possible solution and being shared with us.

I'd love to give it a shot, but I don't have the free time to do something like that. I'm already juggling too many projects as it is.

Please elaborate on your details if I got this wrong, but I read this as being an issue with your app Xcode project file being hard to merge?

Yes, CocoaPods rewrote that project file on every single pod install / pod update. I can accept that this is a bug. However, it still has to rewrite the project in the two scenarios you described. A couple weeks ago a coworker almost committed a CocoaPods-written version of the project file because of adding a new project configuration. All CocoaPods needed to do was set the xcconfig file for the configuration, but it was impossible to see that's what it was trying to do when it rewrote the whole project, and of course the next project change would rewrite the whole project again (when Xcode converts back to its XML format). Thankfully I caught that and made the project file change by hand instead. But even just a single project rewrite can cause hairy merge conflicts when multiple people are making changes to the project file.

Also to note, I found a ton of UUIDs with values (null) in the project file, on both branches.
That does sounds really bad. Can you please open a new ticket that shows steps to reproduce this?

I have no idea how to reproduce this. I think every single project file rewrite was adding more (null) entries, but I didn't try and track it down. After cleaning up the project file by hand, the (null)s never reappeared.

[lilyball]

As predicted, after installing Xcode 7, CocoaPods is now crashing when trying to call the private APIs in Xcode's private frameworks. See CocoaPods/CocoaPods#4151.