Project Queue Stuck Processing

[albupp]

I'm developing an application which uses the PSI to programmatically create
and update projects. Sometimes they include hundreds of tasks and thousands
of associated custom fields, assignments and dependencies.

Smaller updates seem to work, but larger ones appear to be getting stuck in
the queue. I have three jobs which have been pegged at 50% done for several
days, one since last week. All say they are in the processing state of an
update from the PSI.

The 3 projects are visible from the ProjTool, but not the PWA Project
Center. They occupy positions 1,2 & 3 in the queue. From the sequence of
jobs started by my app, I think it's the final one, an update which includes
a flag telling the PSI to publish the project that's getting stuck. I cannot
cancel any of the jobs. Doing so has no apparent effect in the Queue Manager
job listing. Of course stopping and restarting the queue service or
rebooting the server have no effect on this.

At the time that the 1st job entered the queue, I find this error in the
Windows app event log:

Faulting application microsoft.office.project.server.queuing.exe, version
12.0.6211.1000, stamp 46ce70f1, faulting module kernel32.dll, version
5.2.3790.4062, stamp 46264680, debug? 0, fault address 0x0000bee7.

Following this, I get repeated warnings such as this:

Standard InformationSI Entry Point:
Project User: NT AUTHORITY\NETWORK SERVICE
Correlation Id: 2e832c71-775b-4e71-a9b5-02c618ebdc00
PWA Site URL: http://prjsvr01/pwa2
SSP Name: SharedServices1
PSError: Success (0)
Jobs locked by machine 'PRJSVR01' were unlocked and made available to other
running queue services. The queue type (project queue, timesheet queue etc)
which was non-responsive is: 'ProjectQ'. Queue instance UID of the
non-responsive queue is: 'f5211a00-08be-4ea4-92f8-be45683f9e7b'

Have I somehow mangled the queue beyond repair? I'm thinking it's time to
re-install Project Server since it's only a dev host at this point and
doesn't contain any actual user data. Still that would be a hassle if
avoidable since it would mean recreating a number of enterprise custom field
defs.

Any ideas?

Thanks in advance.

 

[Gary L. Chefetz]

Albupp:

I would recommend opening a thread on this topic in the developer's group.
When you say "thousands" of custom fields it sets off a bit of an alarm bell
as you may be hitting some SQL limiatations.

--
----------
Gary L. Chefetz, MVP
MSProjectExperts
Project Server Consulting: http://www.msprojectexperts.com
Project Server Training: http://www.projectservertraining.com
Project Server FAQS: http://www.projectserverexperts.com
Project Server Help Blog: http://www.projectserverhelp.com

 

[albupp]

Gary,

Thanks for your reply. I did some research yesterday and discovered that
there's an Infrastructure Update from 2007 that we hadn't applied to our
server. After doing so, the stuck jobs have become simply failed ones, and I
think the queue is back to a stable state.

I have spent a lot of time working on making large updates to MSP via the
PSI, and you're right have alarms go off when someone talks about doing that
since there are definate gotchas lurking there. I've already come across the
most salient of them, namely the need to break up PSI requests into chunks of
less than 1000 records.

I still don't know what the problem was, and I still see linger and
unexplained things come out of the queue, but the update clearly helped.

Cheers, Albert

 

[Gary L. Chefetz]

More than likely you weren't cancelling jobs stuck in the getting enqueued
state as this checkbox was not selected by default under advanced options on
the manage queue page prior to the Infrastructure Update. After applying the
Infrastructure update, the checkbox is selected by default and these stuck
jobs become visible.

--
----------
Gary L. Chefetz, MVP
MSProjectExperts
Project Server Consulting: http://www.msprojectexperts.com
Project Server Training: http://www.projectservertraining.com
Project Server FAQS: http://www.projectserverexperts.com
Project Server Help Blog: http://www.projectserverhelp.com

 

[albupp]

Actually, that wasn't the case. I tried enabling that option before the
update and it failed to make any difference since the jobs were already in
the queue, and stuck at 50% processed for days. Still the update somehow or
other helped. Judging from the event and trace logs, I'd venture a guess
that as part of the update the queued jobs were traversed and failures
detected that now allowed them to be marked as failed instead of processing.

 

[Sean Hanson]

We handle 50,000 by being chatty with our code, process batch, wait til done,
feed next batch, dont try to stream it all at one shot, batches of 5, trickle
it through, way better.
--
Sean Hanson

http://www.randsmanagement.com
Mass Resource Tool for 2007
Project Server 2007 Archive Tool
2007 ULS Log Reporting