How to Upload a 4,000 Photo iCloud Photo Library into a 40,000 Photo macOS Photos Library to Free Up iCloud Drive Space

Today I discovered that I didn’t have enough iCloud storage space for a file I wanted to share.  In the past (apparently since sometime in 2015), I was able to find space here and there, but today, it appeared, I was out of options.  Actually, I don’t really use iCloud for much, storage-wise.  I upgraded my iCloud storage to the 50G plan, mainly just for backups for apps, email, and a few documents, but with all that extra space (coming from the free 5G plan), I had decided to turn on backups on my other iOS devices and… to turn on the iCloud Photo Library on my phone.  I didn’t really understand how it worked.  Apple always seems to be changing its photo synching services.  I had imagined it would be awfully nice if I didn’t have to physically connect my phone to my computer to upload my latest photos, because up until I activated the iCloud Photo Library, I would occasionally sync and select to remove photos from my phone after import.  So through some wishful thinking, my synching sessions got fewer and fewer and I noticed that it never seemed to free up much space.

Somewhere along the line, iPhoto gave way to the Photos app on macOS, and the “remove after import” checkbox went away and I figured they must have figured it out the way I’d previously imagined.  After each import, the photos would no longer be detected on the phone from the photos app, so I just hoped they were getting transferred.  I was confused that I still seemed to have photos on my phone – lots of them, but I still had space, so I just hoped everything was being magically handled.  Besides, I had other things to worry about, so I chose to ignore the problem and just hope that things were somehow magically and wirelessly getting to my computer.  I willfully put out of my mind and didn’t think about the fact that I had intentionally not turned on the iCloud Photo Library on my computer, because I knew I had 10 times as many photos there as I had on my phone.

Today however, I finally tallied the photos on iCloud, and it was over 20 gigs!  I tried synching my phone again.  I couldn’t tell where all the photos on my phone were being stored.  Why could I transfer some, and they would disappear, yet I had seemingly 3 years of photos on my phone.  Were the photos in the iCloud Photo Library also getting onto my computer?  It looked like some were in both places, but were they all?  I searched for a way to limit the space that photos would take up in iCloud.  I googled to find out whether I could safely delete photos from my phone(/iCloud?) and I got mixed messages.

The thing is, it’s not transparent where a particular photo is stored when I’m in the photos app and there’s no way to let my computer handle it the way it had in the past.  So I decided to recruit some help and called Apple.  I eventually got elevated to a Photos app expert (Jacob), and he explained things very well.  Here’s how I understood it:

  1. All the photos on my phone were being saved on iCloud, and only in iCloud.
  2. I had enabled “Optimize iPhone Storage” in the Photos Settings, which means that I only had compressed/smaller versions of the photos actually on my phone.  The full versions were in iCloud.
  3. When I synched my phone with my computer, I was only getting recent photos that hadn’t uploaded to iCloud yet.
  4. Those transferred photos weren’t removed from the phone, so they also ended up on iCloud (but anything that had uploaded to iCloud since the last time I synched, never made it onto my computer).
  5. The macOS Photos app does not synch the compressed/small versions of photos on my phone that were reduced after uploading to iCloud.

So I was faced with an issue.  Normally, one would just turn on the iCloud Photo Library on the computer and it would download all the high res photos, but doing that would be a 2-way synch and I had 10 times the amount of photos on my computer.  There wasn’t enough space in my iCloud drive to synch everything.  I could have bought more iCloud drive space, but I already have the photos on my laptop backed up on a NAS drive, so I’d prefer to just transfer the iCloud Photo Library to the computer.  So here’s how we did it…

  1. Quit the macOS Photos app.
  2. Start up the macOS Photos app while holding down the option key.
  3. Select to create a new photos library and name it something like “iCloud Photos”.
  4. Open System Preferences -> iCloud.
  5. Turn on Photos and click the options button.
  6. Turn on the iCloud Photo Library.
  7. In macOS Photos, click “Photos” at the top left.
  8. Scroll to the bottom and wait for the downloading message to change to “Updated Just Now”.
  9. Select all the photos and choose Export -> Export Unmodified Original…
  10. Save the photos in a new folder on the desktop.
  11. Quit macOS Photos.
  12. In System Preferences, turn off the iCloud Photo Library.
  13. Start up the macOS Photos app while holding down the option key.
  14. Select the original Photos Library.
  15. Select File -> Import…
  16. Select the folder you created on the desktop and click “Review for Import”.
  17. Wait for all the photos to be detected then select them and click “Import All New Items”.
  18. When the import is done, delete the folder you created on the desktop and the iCloud Photos Library that Photos created in your “Pictures” folder.
  19. [Proceed if you wish to remove the photos from iCloud, and thus your other devices…] Open System Preferences -> iCloud, then…
  20. Click the “manage” button at the bottom right.
  21. Click Photos, then click “Disable and Delete”

I never would have figured this out on my own and I imagine that many of you are just as bewildered as me, that is unless you started taking pictures after the iCloud Photos Library was created.

A couple things to note: 1. Photos you take in other apps (such as in the Messages app, whether they were taken and sent or received) are not in the iCloud Photos Library.  2. The macOS Photos app no longer detects duplicates, so importing photos that you previously synched (before they were uploaded to iCloud) may result in multiple duplicates.  There are apps available on the App Store for detecting and removing duplicates.

Advertisements

Work-around for Zombie Proximitask Reminders

I’ve posted about Proximitask before.  It’s no longer on the app store, despite it’s usefulness (especially when paired with RadBeacons which you can plug into any USB port).  I would post a Proximitask download link if any of them worked anymore, but if you still have it on your iPhone and use it all the time like I do for indoor location based reminders via iBeacons, you may have run into one annoying bug: you mark a reminder as done, but the reminder sometimes persists to pop up.  Here’s a way to permanently quiet those reminders… Continue reading

Using your university’s free Apple Developer account to eliminate the “Unknown Developer” Error in your 3rd-party-developed macOS App

If you work for a university and have developed a macOS app (as a part of your academic job) using 3rd party software (e.g. a Java app developed with Eclipse and built with Gradle), you’re probably familiar with telling your users how to deal with the annoying “Unidentified Developer” error they get when they double-click your app:

negcontrol.png

They get a window saying “<your app> can’t be opened because it is from an unidentified developer“.  The user is given no clear option on how to run the app except a vague reference to “security preferences”.

There are things the user can do to get around this error without you becoming an “identified developer”.  A user can perform a one-time bypass by right-clicking the app and selecting open or they can muck around with their Security & Privacy System Preferences settings.  They can even execute some commands in the Terminal to reveal a setting that has been hidden in the past few versions of macOS to always allow apps from anywhere.  But what would be nicer is to eliminate this error so that the user doesn’t have to do any of that.  Normally that involves giving Apple money to pay for membership in their developer program, but if you’re in academia, Apple has free such accounts for “Educational Institutions”.

Chances are, your university has already enrolled in this program and all you have to do is hook into that resource.  But be aware that universities have an image to uphold and that digital identity extends into the App Store, so they’re not likely to welcome any app to sit alongside their custom campus, reunions, and transit apps.  You can belay their fears about your app by noting that you intend to distribute your app outside of the App Store.  Users who download your app won’t even ever see the university’s name unless you added it to your app yourself.  All you need is a certificate that will allow you to code-sign your app so that Apple can know who you are.

It took me awhile to navigate this system.  I initially ran off on this tangent trying to create a free “education” Apple Developer account by digging up and supplying my University’s D-U-N-S number, official address, etc… and ran into a form that asked me to assert that I could commit the university to legally binding agreements, at which point I realized that I was on the wrong path.  Free educational institution Apple Developer accounts aren’t per person, faculty, or department.  They’re 1 per university.  I already had a couple of official university apps on my iPhone, so I knew there already existed an account, and made a number of calls before I got put in touch with the Apple Developer Account “holder”.

Those in charge of the Apple Developer Account here were only familiar with iOS app development, so they didn’t know how to advise me on accomplishing what I wanted to accomplish.  My sole goal was to eliminate the “Unidentified Developer” error.  No-one knew whether that meant I had to distribute my app on the app store or what, and they were of course concerned about the app showing up alongside all their campus-centric apps.  I managed to convince them that I wouldn’t be distributing on the app store and that I believed all I needed was a code-signing certificate.  Since no-one here knew how to do that, I called Apple again.

Apple really pushes their developers into Apple’s canned development resources (XCode) to do their development.  In fact, even their developer support team members tend not to know anything about code-signing and certificates, because it’s all handled in the background behind XCode.  I had numerous phone calls with Apple Developer Support and they proclaimed that it sounded like I was “speaking a different language”.  They weren’t even familiar with the “Unidentified Developer” error and immediately referred me to the makers of the 3rd party development apps I was using, implying it was a problem with their software.  I persisted however, because I had already tried code-signing using apple’s codesign command line utility and managed to get them to at least confirm that what I suspected was happening was happening.  And that is, that not any certificate will do… but I’m getting ahead of myself.

Here, I will walk you through, in detail, the steps I took to eliminate the error from my latest app (TreeView3)… Continue reading

Today’s Apple Feedback: Underlined Event Day in Mail Wrong When Tap-creating an Event

I’d been victim to this bug and forgetting about work meetings a few times recently – and I knew I’d created an event with a reminder for each forgotten meeting! Luckily the meetings were in my office and I was in the vicinity the couple times it had happened recently (and this tends to happen for Monday meetings most frequently). This morning, Friday May 11, I got a confirmation email for a meeting at 1:30 on Monday.  I underlined that text, because that’s how it appeared in Mail, as you’ll see in the screenshot below.  Apple had detected the event time/day text and underlined it so that I could conveniently create a meeting appointment in my calendar app and I knew it would have my default 1-day prior alert. So I tapped the text and entered the meeting title and location and observed that the default-entered time was from 1:30 to 2:30.  I didn’t check the day!  But I knew that events had gone missing in the past, so after creating the event, I set a reminder to double-check it was still there when I got to work.  I had assumed that the events were somehow getting deleted via some sync issue.

When I got to work, I double-checked my calendar and the event wasn’t there!  I then searched my calendar and discovered that the event had been created for Thursday May 10th instead of the expected Monday May 14th!  And today (when I created the event) is Friday May 11th!  Thus, Today’s Apple (iPhone) Feedback:

Taping underlined event time/day in mail creates event yesterday instead of underlined weekday

On Friday May 11, I created an event via the underlined text: “1:30 on Monday” in Mail on my iPhone 6S. I’d noticed events created on my iPhone going missing in the past, so I created a reminder to check my calendar when I got in. It was missing!, so I searched for it. It had been created for Thursday (yesterday!). I hadn’t touched the day when I created the event. All I had done was add a title & enter a location. I observed the time was 1:30, but hadn’t double-checked the day. I just did it again & it’s still creating the event on Thursday!:

Please fix this. If the underlined text tapped says “Monday”, should be created on Monday by default!

Note, I had pasted URLs to my screenshots because Apple’s feedback form doesn’t allow you to attach images.

I’m guessing that the wrong day possibly came from the fact that the email thread discussing a possible meeting time started on Thursday.  Plus, the first underlined proposal of 1:30 didn’t have a day of the week underlined with it.  So my guess is that that underlined time got associated with the current day (even though it was past 1:30 on Thursday when we started emailing about meeting).  And it seems possible, albeit not a good design decision, that when you tap underlined text to create an event in Mail, that despite the particulars of the specific underlined text, that Apple is re-parsing the entire email to extract what it thinks the intended day & time of the event is likely to be.  It’s just getting it wrong in this case.

If you don’t want to miss meetings because of this but, please consider echoing my Apple Feedback.

Services to sum or subtract highlighted numbers and calculate equations

Years ago, I discovered the convenience of treating the Spotlight search as a calculator.  If you paste in an equation, the top result will be what the calculator app produces given the math you’ve typed or paste in.  I take a lot of notes in my work and frequently have a simple equation to calculate or numbers to sum or subtract.  It’s easy enough to copy and paste an equation into spotlight or paste a set of numbers (with some manual clean-up) to Excel or even search & replace with plus or minus signs & paste into spotlight, but you’d be surprised how much more efficient it feel to do this automatically.  You can even have it paste the result after the end of the highlight.

Combine this with the fact that many text editing apps allow you to select columns of text, and it becomes a trivial matter to do things with masses of numbers in any text and any app.

The sum & subtract automator services also conveniently strip out non-number characters, so you can highlight any numbers that have text in between and sum or subtract them.

Sum Service

Let’s start with the Sum service.  Here’s an example of how it works:

sum_demo.gif

As seen above, you simply:

  1. Select text with the numbers you want to sum
  2. Right-click the selection, and
  3. Select Services->Sum.

The numbers are parsed and added using Spotlight.  In the demo above, I simply tap the escape ket to exit spotlight after seeing the result.  But, you could select & copy the result to use it how you wish, directly from the spotlight result listing.

Subtract Service

The subtract service works the same way as the Sum Service above, but instead of adding all the numbers, it simply takes the first number and subtracts all the rest.  Both services ignore non-numbers and can handle decimals.  It even knows the difference between hyphens in words versus negative numbers.  However, they may have some trouble with non-standard hyphen characters and periods in full sentences.

Calculate (& Append Result) Service(s)

The Calculate Service (or “Calculate & Append Result”) will take any equation that Spotlight (i.e. the Calculator app) can handle and compute the result.  And if you know already that you will want to append the result after the highlighted text in the form of ” = answer”, you can alternatively select the “Calculate & Append Result” Service.  Here is a demo using the service:

calc_demo.gif

Installation

  1. Go to the github gist containing the Applescript code for each service
  2. Copy the code from one of the 3 files in the github gistcopygistcode.png
  3. Open Automator.appautomatordock.png
  4. Select service “Service”/gear icon from the dropdown sheet & click “Choose”selectservice.png
  5. Drag the “Run AppleScript” action into your workflowautomatorapplescript.gif
  6. Replace the purple code in the Run AppleScript action with the code you copied in step 1
  7. Save the workflow and name it however you would like it to appear in the services contextual menu (E.g. “Calculate & Append Result.workflow” – the extension will not appear in the menu)
  8. Repeat for the remaining 3 services.

The workflows/services will be saved automatically to your Library/Services directory in your home directory.  If you right-click the file name at the top of the window in Automator, you can select the Services folder to reveal it in the Finder.  You can then copy that file and send it to any other computer you would like to also have that service.

Have fun!

Disclaimers: These services are only intended as a quick and dirty solution to work in any context, & any app.  If you have a repeated common use-case, consider other solutions.  Note also that any application which reserves the arrow keys for some function when the shift key is held down, other than modifying the most recent selection, this service will fail.  Some applications, such as java applications, modify selections using shift-arrow navigation differently, depending on the direction of the mouse drag during text selection.  This can produce unexpected results.  A work-around for both such issues can be to use the strategy used for Terminal.app, but this would required modification to the code.  A few of the features in the script rely on some tricks such as statically set delays and command-line calls, necessary to either wait for an application to respond or to control the focus of various windows.  If your computer is very busy or has any configuration issues, the proper functioning of these services may be disrupted.  These services were developed and tested on macOS High Sierra, 10.13.4.  They may or may not work in other macOS versions.

Today’s Apple Feedback: High Sierra is a Heavy Sleeper

Since upgrading recently to High Sierra, I’ve been dealing with the fact that my keyboard would no longer wake my computer from the screen saver.  All keyboard presses (other than modifier keys) would only cause the computer to beep (and screen-flash, since I set that in the accessibility preferences).  I finally decided to resolve this issue today, since I am between projects.  It took awhile to dig up a solution via google search-term tinkering, so I thought I would share a work-around I found here.

Briefly, to work-around this issue and get your mac to wake up upon any keyboard press, you must open up System Preferences, click the Mission Control preferences pane, and turn on “Displays have separate spaces”:

mcsepspaces.png

Note, you will have to log out and back in (no restart required though), as is noted in the gray message next to the setting seen in the image above.  After that, the keyboard should start being able to wake your computer.  If you turn this setting back off & log out/in, the problem will recur, so unfortunately, if you have multiple displays, are experiencing this issue, and don’t want them to have separate spaces, you’re out of luck.  I suggest you submit feedback to Apple, as I did below:

Keyboard presses & mouse motion don’t wake from screen saver since High Sierra upgrade

Since installing High Sierra, I have been unable to wake my computer from the screen saver using the keyboard. All keyboard presses result in a beep/screen-flash (since I have that set). A work-around described here works:

https://forums.macrumors.com/threads/screensaver-behavior-in-high-sierra.2075712/

By turning on “Displays have separate spaces” in the Mission Control system prefs, logging out, & back in, the problem no longer occurs. The problem can be made to recur by reversing that process.

I have 10.13.3 (which is not listed below in the select list).

Automator Services for finding coordinates in DNA/AA strings

Do you write code to analyze or modify DNA or proteins?  Do you do your work on a Mac?  If so, I have a few Automator Services, written in AppleScript, that you may find very handy:

  1. Get the sequence (& alignment) length of a selected nucleotide string
  2. Get the sequence length of any selected string (e.g. protein or quality string)
  3. Show where a coordinate is in any selected string (including white-spaces)
  4. Show where a coordinate is in a selected sequence (e.g. protein or quality string)
  5. Show where a nucleotide coordinate is in a selected sequence
  6. Show where an alignment coordinate is in a selected nucleotide sequence
  7. Get the reverse complement of a selected nucleotide sequence
  8. Guess the barcodes present in a FastQ file *NEW

With these, you can highlight a sequence anywhere in any application and either get the selection length or show where a supplied coordinate is in the selected sequence.

Each service, once installed, will show up in the contextual menu that shows up anytime you right-click any selected text, system-wide on your mac, under the services sub-menu, e.g.:

countntsservicemenu.png

Here are the full details of how to use each service and what it does:

1. Get the sequence (& alignment) length of a selected nucleotide string

Name: Count Nucleotides

countntsoutput.png

This service does a bit more than count nucleotides.  As seen in the example on the right, it reports the number of nucleotides in the selection (sequence length), the length of the selected alignment, the number of discrete & ambiguous nucleotides, and a breakdown of all case-insensitive sequence characters (including gaps and ambiguous nucleotides).

Spaces, tabs, newlines, carriage returns, numbers, or any other non-sequence characters are completely ignored, so you can select any sequence, even if it is formatted & displayed with coordinates.  Only the sequence found inside the highlighted text is considered.  The first selected sequence character is coordinate 1.

2. Get the sequence length of any selected string (e.g. protein or quality string)

Name: Count Sequence Characterscountnonwschars.png

This service counts every character selected except for spaces, tabs, newlines, and carriage returns.  It’s good for getting the length of selected unaligned protein sequences (with no formatted coordinates in the selection) or of quality strings.  Note, there is currently no service for aligned amino acid sequences, but if you would like such a service, let me know in the comments and I’ll whip one up.  I work mostly with DNA, thus I haven’t had much need for aligned protein coordinate determinations.

3. Show where a coordinate is in any selected string (including white-spaces)

Name: Select N Characters

This service works only on “solid sequence” (i.e. having no whitespaces, hard returns, or for that matter: any non-sequence characters).  See services 4-6 for sequence-specific functions.  The way this service shows where a coordinate is, is by changing the length of the selection.  The resulting last character of the selection after the length modification is the length supplied by you, the user.  For example, if you tell it to select 4 characters in this string you selected: ATGCCGTAG, the selection will end up as: ATGCCGTAG.

lengthprompt.png

There are 2 ways to supply the coordinate.  The default way is to grab the coordinate from the clipboard, so all you have to do is copy the number you want to use to set the selection length.  However, if the content of the clipboard is not a number, a popup window will appear to ask you to enter the desired selection length.

 

To use this service:

  1. [Optional] Copy a number/length indicating the amount of the sequence you want to select.
  2. Select any length of sequence from the start position (position 1)
  3. Right-click the selection and select Services -> Select N Characters
  4. [If you didn’t do step 1] Enter the length of sequence you want to select

This script alters the selection length of the selected text you right-clicked on in the window in which you clicked on the sequence, regardless of application.  However, if you are in Terminal.app, it displays the selection result in a popup window instead of in the Terminal itself.  This is because the modification of the selection length is accomplished by shift-arrow keystroke emulation and this is not a means by which you can modify a selection in the Terminal app.  This has 1 side effect.  Normally, if the entered length is longer than the selection, in any other app, that’s not a problem and the selection just expands, but in Terminal’s popup result window, all the service has access to is the selected text, so placeholder ‘N’s are appended to show the desired sequence length.

lengthresult-terminal.pngThere are other drawbacks to this Terminal selection-length work-around.  The font is not fixed-width and the width of the popup window is fixed at a fairly narrow size, thus large sequences cannot be displayed very well.

Since the selection modification happens via emulated keystrokes, it takes a little time for the final selection to be made, but you’ll see how fast it goes as you watch the selection being made.  Since it’s not instantaneous, the script will either adjust the selection from the end or select anew from the beginning for efficiency.

4-8. New services

Names: Select N Sequence Characters, Select N Nucleotides, Select N Alignment Characters, Reverse Complement, & Guess barcodes

These services operate just like Select N Characters, but take the character type into account.

Select N Sequence Characters doesn’t include whitespace characters such as spaces, tabs, and newlines in the calculation of a coordinate in a string.

Select N Nucleotides doesn’t include non-nucleotide characters such as white spaces, gap characters, numbers, etc. in the calculation of a coordinate in a string.

Select N Alignment Characters behaves just like Select N Nucleotides but includes gap characters in the calculation of a coordinate in a string as the one exception.

Guess Barcodes allows you to right-click on a file and find out what the likely barcodes are.  It takes a little bit to run and makes some common assumptions, but if you think the results are wrong, you are given the opportunity to tweak the parameters and run again.

Installation

  1. Go to the github gist containing the Applescript code for each service
  2. Copy the code from one of the 3 files in the github gistcopygistcode.png
  3. Open Automator.appautomatordock.png
  4. Select service “Service”/gear icon from the dropdown sheet & click “Choose”selectservice.png
  5. Drag the “Run AppleScript” action into your workflowautomatorapplescript.gif
  6. Replace the purple code in the Run AppleScript action with the code you copied in step 1
  7. Save the workflow and name it however you would like it to appear in the services contextual menu (E.g. “Count Non-Whitespace Characters.workflow” – the extension will not appear in the menu)
  8. Repeat for the remaining 6 services.

The workflows/services will be saved automatically to your Library/Services directory in your home directory.  If you right-click the file name at the top of the window in Automator, you can select the Services folder to reveal it in the Finder.  You can then copy that file and send it to any other computer you would like to also have that service.

Just try your new services out by right-clicking on selected text anywhere.selectncharsexample.gif

And as you can see from the example above, Select N Characters works on any text.

Have fun!

Disclaimers: These services are only intended as a quick and dirty solution to work in any context, & any app.  If you have a repeated common use-case, consider other solutions.  Note also that any application which reserves the arrow keys for some function when the shift key is held down, other than modifying the most recent selection, this service will fail.  Some applications, such as java applications, modify selections using shift-arrow navigation differently, depending on the direction of the mouse drag during text selection.  This can produce unexpected results.  A work-around for both such issues can be to use the strategy used for Terminal.app, but this would required modification to the code.  A few of the features in the script rely on some tricks such as statically set delays and command-line calls, necessary to either wait for an application to respond or to control the focus of various windows.  If your computer is very busy or has any configuration issues, the proper functioning of these services may be disrupted.  These services were developed and tested on macOS Sierra, 10.12.6.  They may or may not work in other macOS versions.