History for convert CMFPhoto to AT
For future reference....
changed:
-
<div style="width:100%"><pre style="white-space:pre-wrap">From: Jamie Robe - view profile
Date: Thurs, Jul 27 2006 12:40 pm
Not yet rated
show options
Hi. I am updating this thread on CMFPhoto and CMFPhotoAlbum migration or
conversion from the Plone 2.0.5 (with CMFPhoto and CMFPhotoAlbum
products) to Plone 2.1.3 or 2.5. I finally was able to get everything
over using a script that I worked on (started with one I found that had
to do with ZphotoSlide products - see my acknowledgements and thanks in
the script below). I hope anyone googling for help on this issue will
find this as a starting point :-)
The script should be run on a test "sandbox" with a copy of your
Data.fs, not your production server/Data.fs. There are two types of
errors that I had to contend with as I finally ran my conversion:
1) MemoryError = I basically made a copy of my 2.0.5 Data.fs, with my
plone site in it. When I got this working on my test server (all
Products installed, looked just like my real site), I made a second
plone site using the ZMI, installed the script in that plone site, and
then did a simple copy/paste of the albums that needed conversion. When
I tried to do the whole thing at once, it would run for a long time and
then I would get a out of memory type error. So I had to do 2 things. I
copy/pasted the major subalbums over in chunks (lucky for me my stuff
was organized with 5 or 6 major subalbums), and then ran the script each
time for each chunck. If that didn't work, I found I could uncomment out
a "return" in the main loop of my script (see my comments in the script)
that esentially let the script only run one album at a time. Once it
spit out the debugging printout for that album (normal output to let you
know what happened), just hit the back key and then hit "test" tab to
run - it will do the next. IT seemed to help with some really huge photo
albums. I had many layers of nested albums, and it works for all of
that.
2) On my last major part of my site that I ran this on, I started
getting a "BadRequest" error page!!!! It listed an id of "CPA 06-22_old
does not exist". My script make a temporary version of each album or
photo with a _old tacked on the end of the id, so I knew it was probably
a problem with the id. Sure enough, someone had typed in bad shorts
names on some items, in this case it had a space and a dash. I just went
in and did a search from the copy of my site, and fixed the shortnames.
Thanks goodness it was ony about a dozen :-) Ran the script again and it
worked fine. Guess 2.0.5 could handle even badly typed in shortnames,
while the python script or Zope commands chocked when it hit those. I
researched and found who had done this by mistake...
Remember, this just gets turns your albums into folder and your photos
into images (all in 2.0.5) - you then have to migrate your stuff in the
normal way to the 2.1.x or 2.5 Plone version. The folders in the higher
levels of Plone have album views built-in. I am sure I have some manual
tweaking and some redo of user permissions, but it saved me form losing
all my titles, descriptions, and such if I had to use WebDav. I learned
from this experience, and you may be able to modify this approach for
conversion of other content.
I am pasting the code that worked for me below - see my comments I have
included in the script. I know it is not optiomized, but it did work for
me and help me get thousands of photos and albums convereted over with
titles, descriptions, etc.
########################################################################
#######
# This script is designed to process photo album and photo objects that
use CMF Photo and CMF Photo Album
# products in Plone 2.0.5, "converting" them to new folder and image
objects, but retaining the
# original ids, title, descriptions, and contents. It can process nested
photo albums, and photo albums
# with objects other than the photo type objects. You will end up with a
plone site that has all photo albums
# changed into folders, and all photos changed into images. You can then
do a normal migration to
# Plone 2.1.x or 2.5.x in the normal way. These versions do not support
the CMF Photo and Album products, and
# the migration script never worked for me.
#
# I hope this helps someone else like me migrate to the new plone, if
you have thousands of albums and photos
# in the old Plone. Webdav is a fallback way of migrating, but I had 2
years worth of titles and
# descriptions typed into my content, and did not want to lose that when
webdav just writes folders and files
# using the often cryptic photoalbum.xxxxxxxxxxxxxxxx type name on the
file system disk. Plus, I always seem to
# get errors when doing webdav on huge numbers of image files, which
usually stops the process in the middle
# somewhere. I started this script based on a script I found by Morten
W. Petersen, which he wrote to convert
# ZPhotoSlides, etc. There are many changes, I presume based in
differences with the various products and versions
# involved. I also got ideas and help on this Plone-Users email list -
Thanks to everyone! I offer this as-is,
# back to the list on my original thread, in case someone needing this
finds it in the future.
#
# To run this script, make a copy of your Data.fs and transfer it to a
non-production machine, running
# Plone 2.0.5 and any applicable products, including the CMF Photo and
CMF Photo Album products. You
# then go to your ZMI, and in the root of your site, add a script
(python). Click the add and edit button
# as you are not uploading a file, you can simply do a cut and paste of
this code.
# Make sure that the indenting is correct, as sometimes the spacing gets
messed up and this will
# break the script. Python uses the indenting to keep track of what code
is in a loop or if statement.
# Once you have the script code pasted in, click save. If everything
went in correctly, you will not get
# any errors. If you have syntax errors, check your indenting!
# I recommend creating a plone site in the same Data.fs, with the same
products installed as your
# original, but only a few photo albums (some nested inside other
albums) and photos for testing.
# It also helps by speeding up testing of the script and any changes, as
you can delete the test folder, with
# whatever good or bad results in it :-), and then get a fresh test
folder using copy and paste in the ZMI from
# the one plone site to the other. The scripts runs against only the
plone site it is sitting in!
# To run the script, clck on the Test tab at the top pf the script page.
It should run and print out the
# results of the debugging print lines you can see int he code below.
# If you have problems, the first thing to do is to look at the code at
the bottom of the script, where
# it loops thru a list of all the photo related objects in your plone
site. You can uncomment out the
# line that says "return printed", in the main inner loop, and this
should cause the loop to process just
# one album and then stop and return a report to be printed after the
convert(....) subroutine is run.
# Note that I have tried to comment what is happening, but be careful
and very systematic if you make
# modifications - change only a line or command one at a time. There are
lots of objects, object ids,
# with lots of renaming, cutting, pasting, and deleting going on.
# Note that the most problematic part of this was the deletion of the
old albums and old photos after
# those objects had new counterparts (folders and images) created. The
problems stemmed from the way
# manage_delObjects works on lists of object ids, and that when I
deleted an object like an old album,
# even though I had renamed its id with a _old at the end, it would mess
up any higher level loops going thru
# say, all the ojects of type album. I had a curious effect where it
would skip every other id. So I had to
# make old photo deletion after the loop, using a list built-up during
the processing of those photos into images.
# Same with the albums, and that entailed changing the main loop at the
bottom of this code, so that it
# used a while loop, with regeneration of the list after each one was
processed, to avoid errors.
#
# I have left i some various debugging lines, most commented out with #.
Just to show you how to stop
# the processing at various points and how to get some info printed to
your screen. You should end up with
# a screen in the ZMI that is just the result of your various print
statements in the code. Note that I had
# build up some text reports in a variable to pass back to the main
loop, in order to avoid a printing error.
# You can alays add in a line like: return "Got to point B" or something
like that, to see if you can figure
# out any bugs. It runs for me, but versions and differences with your
software could cause bugs I suppose.
# My production websites all run on Debianb Linux boxes, but my Intranet
sites run in Windows versions or 2.0.5.
# I did all this coding and debugging on a clean Windows install of
2.0.5, but I think it would work on Linux as
# well. Just do not run this on your production box or production
Data.fs. Go to plone.org and install it on any
# PC and work on copies of everything.
#
# Be sure to go to portal_types in the ZMI and go to the photo album
type and uncheck the
# box filter content types - this is in case the arbitrary order of any
nesting requires the
# processing of a photo album object (making a new folder with its id,
title, etc and converting
# photo objects and then cut and pasting any other folders or albums or
other types) inside a parent
# that happens to be a photo album.
#
# My name is Jamie Robe and my email is robej(at)plancom.org. I suggest
asking questions about this on the
# Plone-Users email list, as other people probably have more answers
than I will on specific problems. But what
# I have contributed here did work for me. Good luck!
# This subroutine processes each photo album one at a time, as called by
the main loop.
def convert(thealbum):
# the following checks if it is a photo or album and assigns the id,
# which is the "short_name" of the object (not title)
if callable(thealbum.id):
# these are photos
thealbum_id = thealbum.id()
else:
# these are albums
thealbum_id = thealbum.id
# check if thealbum is really still an album(hasn't been converted
since script's main loop started)
if thealbum.meta_type == 'Photo Album':
# continue doing conversion steps of this object
# next we check the type of parent object it is sitting in
thealbumparent = thealbum.getParentNode()
if thealbumparent.meta_type == 'Photo Album':
# sitting in unprocessed photo album so convert parent first
# the following is commented out, as we do not need
recursive process here
# as it was messing up the loop
####convert(thealbumparent)
# note the nothingatall=0 are just code place holders for
indents, they do nothing :-)
nothingatall=0
else:
# sitting in ok folder so don't convert parent
nothingatall=0
# now that we are not in a photo album parent do rest of the
steps to this album and its contents
# rename original object for prep for info transfer then
eventual deletion of this old one
thealbumparent.manage_renameObject(thealbum_id, thealbum_id +
'_old')
# create a new folder object with original thealbum id
portal_types = thealbumparent.portal_types
# note that the following transfers over the title and
description from original thealbum object
portal_types.constructContent('Folder', thealbumparent,
thealbum_id, None, title=thealbum.title,
description=thealbum.description)
# for the old album just use the thealbum, since object same
even if id was changed with _old
##old_album = thealbumparent[thealbum_id + '_old']
new_folder = thealbumparent[thealbum_id]
# now loop thru contents of the original album object that are
photos only and
# make new images in new folder with corresponding data, titles,
etc
# note that we are accumulating a list of ids to delete after
the loop, for a one time deletion to avoid
# errors that happened when delete in old photos one by one in
the loop
# thereport is for debugging purposes only, as it can be passed
back and return printed as needed.
thereport =''
killlist=[]
for thephoto in thealbum.objectValues('Photo'):
if callable(thephoto.id):
thephoto_id = thephoto.id()
else:
thephoto_id = thephoto.id
if thealbumparent.meta_type <> 'Photo Album':
# just double check and only do photo not photo album
thereport=thereport+"tpid="+thephoto_id
thealbum.manage_renameObject(thephoto_id, thephoto_id +
'_old')
new_folder.invokeFactory('Image', thephoto_id,
title=thephoto.title, description=thephoto.description,
file=thephoto.data)
#killstring="pic3"
killstring= thephoto_id + '_old'
thereport=thereport+"ks="+killstring
killone=[killstring]
killlist=killlist+killone
#now go back and delete all the _old photos from the photo album
thealbum.manage_delObjects(killlist)
# following could be uncommented for debugging for example
#return killlist
# this section builds a list of all ids of objects remaining in
album for cut and paste
# to the new folder, including such things as folders or other
albums or misc files
thereport =''
cutpastelist=[]
for theother in thealbum.objectValues():
if callable(theother.id):
theother_id = theother.id()
else:
theother_id = theother.id
thereport=thereport+"otherid="+theother_id
cutpasteone=[theother_id]
cutpastelist=cutpastelist+cutpasteone
#now cut and then paste all other content from album to new
folder, after the loop
cut_info = thealbum.manage_cutObjects(cutpastelist)
new_folder.manage_pasteObjects(cut_info)
##return thereport
# everything should be converted or copied to an new folder so
delete the album with the modified id
thealbumparent.manage_delObjects([thealbum_id + '_old'])
else:
# was not a photo album, so must have been fixed (converted)
already since the loop started
nothingatall=0
return thereport
# main part of the script that starts up when you hit the test tab, and
it calls routine above
# It starts looping thru all the albums till they and all contents are
processed.
print "starting"
# metatypes=Photo is just CMFPhotos, but Photo Album is both photos and
albums
# load up all the objects that match these photo related metatypes
photostuff = context.ZopeFind(context,obj_metatypes=('Photo Album'),
search_sub=1)
# following was a print for debug to see what objects were going to be
looped thru, and please
# note that I could never get the ZopeFind to just return the albums, it
always included photos too :-(
#print "photostuff1st=",photostuff
while photostuff <> []:
for (path, photos) in photostuff:
#following line is for debugging only, will show all photos and
albums
##print '--meta_type--', photos.meta_type, '--path--',
path,'--name--', photos.subject, '--content_type--',
photos.content_type, '--id--',photos.id,'--title--', photos.title,
'--description--', photos.description,'--photosobj--', photos
# skip all the photos, just do the albums
if photos.meta_type == 'Photo Album':
print "It is an Album so process it"
print '--meta_type--', photos.meta_type, '--path--',
path,'--name--', photos.subject, '--content_type--',
photos.content_type, '--id--',photos.id,'--title--', photos.title,
'--description--', photos.description,'--photosobj--', photos
print convert(photos)
# note that we deleted old album in the convert process, so
need to reload the following again or
# the looping gets out of sync - you will get errors if you
cut and pasted objects
photostuff = context.ZopeFind(context,obj_metatypes=('Photo
Album'), search_sub=1)
# Following is another debug print example, to see the
regenerated list of objects to process (minus the first one hopefully
deleted)
#print "photostuff2nd=",photostuff
# Following is optional breakpoint with print of debugging
print outs
# If you get memoryerror or indications that your PC memory
is low, just uncomment out
# the following line and manually run the scrfipt over and
over, one album at a time.
#return printed
# The following break gets out of the for loop, so that the
list of objects can be regenerated
# to compensate for anything deleted in the convert
subroutine
break
print "finshed"
# The following prints out to the ZMI screen anything printed in the
main part of the script, or
# anything passed back in the convert() subroutine. At first I thought I
would have to keep
# the recursive structure of the zphotosslides scripts, but turned out
this did not need to be
# done here, so probably could have done this without using a
subroutine...
return printed
########################################################################
######
</pre></div>
From pf Fri Jul 28 12:01:53 -0500 2006
From: pf
Date: Fri, 28 Jul 2006 12:01:53 -0500
Subject: For future reference....
Message-ID: <20060728120153-0500@www.fraterdeus.com>
Looks like a good model for other things as well!
Thanks Jamie!
