From:		 Jamie Robe - view profile
Date:		Thurs, Jul 27 2006 12:40 pm
Not yet rated
 
show options

Hi. I am updating this thread on CMFPhoto? and CMFPhotoAlbum? migration or 
conversion from the Plone 2.0.5 (with CMFPhoto? and CMFPhotoAlbum? 
products) to Plone 2.1.3 or 2.5. I finally was able to get everything 
over using a script that I worked on (started with one I found that had 
to do with ZphotoSlide? products - see my acknowledgements and thanks in 
the script below). I hope anyone googling for help on this issue will 
find this as a starting point :-) 
The script should be run on a test "sandbox" with a copy of your 
Data.fs, not your production server/Data.fs.  There are two types of 
errors that I had to contend with as I finally ran my conversion: 
1) MemoryError? = I basically made a copy of my 2.0.5 Data.fs, with my 
plone site in it. When I got this working on my test server (all 
Products installed, looked just like my real site), I made a second 
plone site using the ZMI, installed the script in that plone site, and 
then did a simple copy/paste of the albums that needed conversion. When 
I tried to do the whole thing at once, it would run for a long time and 
then I would get a out of memory type error. So I had to do 2 things. I 
copy/pasted the major subalbums over in chunks (lucky for me my stuff 
was organized with 5 or 6 major subalbums), and then ran the script each 
time for each chunck. If that didn't work, I found I could uncomment out 
a "return" in the main loop of my script (see my comments in the script) 
that esentially let the script only run one album at a time. Once it 
spit out the debugging printout for that album (normal output to let you 
know what happened), just hit the back key and then hit "test" tab to 
run - it will do the next. IT seemed to help with some really huge photo 
albums.  I had many layers of nested albums, and it works for all of 
that. 
2) On my last major part of my site that I ran this on, I started 
getting a "BadRequest?" error page!!!! It listed an id of "CPA 06-22_old 
does not exist". My script make a temporary version of each album or 
photo with a _old tacked on the end of the id, so I knew it was probably 
a problem with the id. Sure enough, someone had typed in bad shorts 
names on some items, in this case it had a space and a dash. I just went 
in and did a search from the copy of my site, and fixed the shortnames. 
Thanks goodness it was ony about a dozen :-) Ran the script again and it 
worked fine. Guess 2.0.5 could handle even badly typed in shortnames, 
while the python script or Zope commands chocked when it hit those. I 
researched and found who had done this by mistake... 

Remember, this just gets turns your albums into folder and your photos 
into images (all in 2.0.5) - you then have to migrate your stuff in the 
normal way to the 2.1.x or 2.5 Plone version. The folders in the higher 
levels of Plone have album views built-in. I am sure I have some manual 
tweaking and some redo of user permissions, but it saved me form losing 
all my titles, descriptions, and such if I had to use WebDav?. I learned 
from this experience, and you may be able to modify this approach for 
conversion of other content. 
I am pasting the code that worked for me below - see my comments I have 
included in the script. I know it is not optiomized, but it did work for 
me and help me get thousands of photos and albums convereted over with 
titles, descriptions, etc. 

######################################################################## 
####### 
# This script is designed to process photo album and photo objects that 
use CMF Photo and CMF Photo Album 
# products in Plone 2.0.5, "converting" them to new folder and image 
objects, but retaining the 
# original ids, title, descriptions, and contents. It can process nested 
photo albums, and photo albums 
# with objects other than the photo type objects. You will end up with a 
plone site that has all photo albums 
# changed into folders, and all photos changed into images. You can then 
do a normal migration to 
# Plone 2.1.x or 2.5.x in the normal way. These versions do not support 
the CMF Photo and Album products, and 
# the migration script never worked for me. 
# 
# I hope this helps someone else like me migrate to the new plone, if 
you have thousands of albums and photos 
# in the old Plone. Webdav is a fallback way of migrating, but I had 2 
years worth of titles and 
# descriptions typed into my content, and did not want to lose that when 
webdav just writes folders and files 
# using the often cryptic photoalbum.xxxxxxxxxxxxxxxx type name on the 
file system disk. Plus, I always seem to 
# get errors when doing webdav on huge numbers of image files, which 
usually stops the process in the middle 
# somewhere. I started this script based on a script I found by Morten 
W. Petersen, which he wrote to convert 
# ZPhotoSlides?, etc. There are many changes, I presume based in 
differences with the various products and versions 
# involved. I also got ideas and help on this Plone-Users email list - 
Thanks to everyone! I offer this as-is, 
# back to the list on my original thread, in case someone needing this 
finds it in the future. 
# 
# To run this script, make a copy of your Data.fs and transfer it to a 
non-production machine, running 
# Plone 2.0.5 and any applicable products, including the CMF Photo and 
CMF Photo Album products. You 
# then go to your ZMI, and in the root of your site, add a script 
(python). Click the add and edit button 
# as you are not uploading a file, you can simply do a cut and paste of 
this code. 
# Make sure that the indenting is correct, as sometimes the spacing gets 
messed up and this will 
# break the script. Python uses the indenting to keep track of what code 
is in a loop or if statement. 
# Once you have the script code pasted in, click save. If everything 
went in correctly, you will not get 
# any errors. If you have syntax errors, check your indenting! 
# I recommend creating a plone site in the same Data.fs, with the same 
products installed as your 
# original, but only a few photo albums (some nested inside other 
albums) and photos for testing. 
# It also helps by speeding up testing of the script and any changes, as 
you can delete the test folder, with 
# whatever good or bad results in it :-), and then get a fresh test 
folder using copy and paste in the ZMI from 
# the one plone site to the other. The scripts runs against only the 
plone site it is sitting in! 
# To run the script, clck on the Test tab at the top pf the script page. 
It should run and print out the 
# results of the debugging print lines you can see int he code below. 
# If you have problems, the first thing to do is to look at the code at 
the bottom of the script, where 
# it loops thru a list of all the photo related objects in your plone 
site. You can uncomment out the 
# line that says "return printed", in the main inner loop, and this 
should cause the loop to process just 
# one album and then stop and return a report to be printed after the 
convert(....) subroutine is run. 
# Note that I have tried to comment what is happening, but be careful 
and very systematic if you make 
# modifications - change only a line or command one at a time. There are 
lots of objects, object ids, 
# with lots of renaming, cutting, pasting, and deleting going on. 
# Note that the most problematic part of this was the deletion of the 
old albums and old photos after 
# those objects had new counterparts (folders and images) created. The 
problems stemmed from the way 
# manage_delObjects works on lists of object ids, and that when I 
deleted an object like an old album, 
# even though I had renamed its id with a _old at the end, it would mess 
up any higher level loops going thru 
# say, all the ojects of type album. I had a curious effect where it 
would skip every other id. So I had to 
# make old photo deletion after the loop, using a list built-up during 
the processing of those photos into images. 
# Same with the albums, and that entailed changing the main loop at the 
bottom of this code, so that it 
# used a while loop, with regeneration of the list after each one was 
processed, to avoid errors. 
# 
# I have left i some various debugging lines, most commented out with #. 
Just to show you how to stop 
# the processing at various points and how to get some info printed to 
your screen. You should end up with 
# a screen in the ZMI that is just the result of your various print 
statements in the code. Note that I had 
# build up some text reports in a variable to pass back to the main 
loop, in order to avoid a printing error. 
# You can alays add in a line like: return "Got to point B" or something 
like that, to see if you can figure 
# out any bugs. It runs for me, but versions and differences with your 
software could cause bugs I suppose. 
# My production websites all run on Debianb Linux boxes, but my Intranet 
sites run in Windows versions or 2.0.5. 
# I did all this coding and debugging on a clean Windows install of 
2.0.5, but I think it would work on Linux as 
# well. Just do not run this on your production box or production 
Data.fs. Go to plone.org and install it on any 
# PC and work on copies of everything. 
# 
# Be sure to go to portal_types in the ZMI and go to the photo album 
type and uncheck the 
# box filter content types - this is in case the arbitrary order of any 
nesting requires the 
# processing of a photo album object (making a new folder with its id, 
title, etc and converting 
# photo objects and then cut and pasting any other folders or albums or 
other types) inside a parent 
# that happens to be a photo album. 
# 
# My name is Jamie Robe and my email is robej(at)plancom.org. I suggest 
asking questions about this on the 
# Plone-Users email list, as other people probably have more answers 
than I will on specific problems. But what 
# I have contributed here did work for me. Good luck! 

# This subroutine processes each photo album one at a time, as called by 
the main loop. 
def convert(thealbum): 
    # the following checks if it is a photo or album and assigns the id, 
    #   which is the "short_name" of the object (not title) 
    if callable(thealbum.id): 
        # these are photos 
        thealbum_id = thealbum.id() 
    else: 
        # these are albums 
        thealbum_id = thealbum.id 
    # check if thealbum is really still an album(hasn't been converted 
since script's main loop started) 
    if thealbum.meta_type == 'Photo Album': 
        # continue doing conversion steps of this object 
        # next we check the type of parent object it is sitting in 
        thealbumparent = thealbum.getParentNode() 
        if thealbumparent.meta_type == 'Photo Album': 
            # sitting in unprocessed photo album so convert parent first 
            # the following is commented out, as we do not need 
recursive process here 
            # as it was messing up the loop 
            ####convert(thealbumparent) 
            # note the nothingatall=0 are just code place holders for 
indents, they do nothing :-) 
            nothingatall=0 
        else: 
            # sitting in ok folder so don't convert parent 
            nothingatall=0 
        # now that we are not in a photo album parent do rest of the 
steps to this album and its contents 
        # rename original object for prep for info transfer then 
eventual deletion of this old one 
        thealbumparent.manage_renameObject(thealbum_id, thealbum_id + 
'_old') 

        # create a new folder object with original thealbum id 
        portal_types = thealbumparent.portal_types 
        # note that the following transfers over the title and 
description from original thealbum object 
        portal_types.constructContent('Folder', thealbumparent, 
thealbum_id, None, title=thealbum.title, 
description=thealbum.description) 
        # for the old album just use the thealbum, since object same 
even if id was changed with _old 
        ##old_album = thealbumparent[thealbum_id + '_old']? 
        new_folder = thealbumparent[thealbum_id]? 

        # now loop thru contents of the original album object that are 
photos only and 
        # make new images in new folder with corresponding data, titles, 
etc 
        # note that we are accumulating a list of ids to delete after 
the loop, for a one time deletion to avoid 
        # errors that happened when delete in old photos one by one in 
the loop 
        # thereport is for debugging purposes only, as it can be passed 
back and return printed as needed. 
        thereport ='' 
        killlist=[] 
        for thephoto in thealbum.objectValues('Photo'): 
            if callable(thephoto.id): 
                thephoto_id = thephoto.id() 
            else: 
                thephoto_id = thephoto.id 
            if thealbumparent.meta_type <> 'Photo Album': 
                # just double check and only do photo not photo album 
                thereport=thereport+"tpid="+thephoto_id 
                thealbum.manage_renameObject(thephoto_id, thephoto_id + 
'_old') 
                new_folder.invokeFactory('Image', thephoto_id, 
title=thephoto.title, description=thephoto.description, 
file=thephoto.data) 
                #killstring="pic3" 
                killstring= thephoto_id + '_old' 
                thereport=thereport+"ks="+killstring 
                killone=[killstring]? 
                killlist=killlist+killone 
        #now go back and delete all the _old photos from the photo album 
        thealbum.manage_delObjects(killlist) 
        # following could be uncommented for debugging for example 
        #return killlist 

        # this section builds a list of all ids of objects remaining in 
album for cut and paste 
        # to the new folder, including such things as folders or other 
albums or misc files 
        thereport ='' 
        cutpastelist=[] 
        for theother in thealbum.objectValues(): 
            if callable(theother.id): 
                theother_id = theother.id() 
            else: 
                theother_id = theother.id 
            thereport=thereport+"otherid="+theother_id 
            cutpasteone=[theother_id]? 
            cutpastelist=cutpastelist+cutpasteone 
        #now cut and then paste all other content from album to new 
folder, after the loop 
        cut_info = thealbum.manage_cutObjects(cutpastelist) 
        new_folder.manage_pasteObjects(cut_info) 
        ##return thereport 

        # everything should be converted or copied to an new folder so 
delete the album with the modified id 
        thealbumparent.manage_delObjects([thealbum_id + '_old']?) 
    else: 
        # was not a photo album, so must have been fixed (converted) 
already since the loop started 
        nothingatall=0 
    return thereport 

# main part of the script that starts up when you hit the test tab, and 
it calls routine above 
# It starts looping thru all the albums till they and all contents are 
processed. 
print "starting" 
# metatypes=Photo is just CMFPhotos?, but Photo Album is both photos and 
albums 
# load up all the objects that match these photo related metatypes 
photostuff = context.ZopeFind?(context,obj_metatypes=('Photo Album'), 
search_sub=1) 
# following was a print for debug to see what objects were going to be 
looped thru, and please 
# note that I could never get the ZopeFind? to just return the albums, it 
always included photos too :-( 
#print "photostuff1st=",photostuff 
while photostuff <> []: 
    for (path, photos) in photostuff: 
        #following line is for debugging only, will show all photos and 
albums 
        ##print '--meta_type--', photos.meta_type, '--path--', 
path,'--name--', photos.subject, '--content_type--', 
photos.content_type, '--id--',photos.id,'--title--', photos.title, 
'--description--', photos.description,'--photosobj--', photos 
        # skip all the photos, just do the albums 
        if photos.meta_type == 'Photo Album': 
            print "It is an Album so process it" 
            print '--meta_type--', photos.meta_type, '--path--', 
path,'--name--', photos.subject, '--content_type--', 
photos.content_type, '--id--',photos.id,'--title--', photos.title, 
'--description--', photos.description,'--photosobj--', photos 
            print convert(photos) 
            # note that we deleted old album in the convert process, so 
need to reload the following again or 
            # the looping gets out of sync - you will get errors if you 
cut and pasted objects 
            photostuff = context.ZopeFind?(context,obj_metatypes=('Photo 
Album'), search_sub=1) 
            # Following is another debug print example, to see the 
regenerated list of objects to process (minus the first one hopefully 
deleted) 
            #print "photostuff2nd=",photostuff 
            # Following is optional breakpoint with print of debugging 
print outs 
            # If you get memoryerror or indications that your PC memory 
is low, just uncomment out 
            # the following line and manually run the scrfipt over and 
over, one album at a time. 
            #return printed 
            # The following break gets out of the for loop, so that the 
list of objects can be regenerated 
            # to compensate for anything deleted in the convert 
subroutine 
            break 
print "finshed" 
# The following prints out to the ZMI screen anything printed in the 
main part of the script, or 
# anything passed back in the convert() subroutine. At first I thought I 
would have to keep 
# the recursive structure of the zphotosslides scripts, but turned out 
this did not need to be 
# done here, so probably could have done this without using a 
subroutine... 
return printed 

######################################################################## 
######