2007 Corrupt reporting database

[Mike Mahoney]

I have a 2 level tsk custom code (long list) which I need to change
substantially. In 2003 I would adopt a cautious approach to such changes. In
2007 making a significant number of changes in a lookup table eventually
cause a problem (unexpected error). Because of this I did an administrative
restore of Enterprise custom fields.
Along the way I get "Reporting Database Refresh Waiting to be Processed
(Sleeping)" appearing in queue. After a considerable period of time this job
fails (Q: how do you force a sleeping job?).
I also have the impression the reporting database is now corrupt. Trying to
build a cube confirms this.
I deleted all published plans and republished one of them and ran the cube
it failed again Error message:

Standard InformationSI Entry Point:
Project User: CHEYNE\mmahoney
Correlation Id: 577e0738-d808-47ab-bede-374c00869e8c
PWA Site URL: http://ccm-devprj01/PWA
SSP Name: SharedServices1
PSError: CBSMetadataProcessingFailure (17005)
Cube build internal error. Fail to create the internal representation of the
OLAP database structure. CBS queued message: Setting
UID=00007829-4392-48b3-b533-5a5a4797e3c9 ASServerName=ccm-devprj01
ASDBName=PWADev ASExtraNetAddress=ccm-devprj01 RangeChoice=0 PastNum=1
PastUnit=0 NextNum=1 NextUnit=0 FromDate=02/19/2007 13:07:44
ToDate=02/19/2007 13:07:44 HighPriority=True. Error: Invalid call to
GetJoinForCustomFieldDimension. Cannot join "Task Non Timephased" fact with a
custom field of entity Assignment

How can I fix the reporting database? How can I further debug this problem?

Thanks

Mike

 

[Mike Mahoney]

Follow up:

The way forward is to remove custom configuration elements from the cube and
rebuild a base cube. This "resets" the reporting database removing any
problem data. After which add back custom fields and rebuild the cube, and
the cube will build successfully.

regards

Mike

 

[Sharry Heberer [MSFT]]

I see that you have resolved your issue, but I just wanted to point out a
couple of things in case you run into this again.

First, and most importantly, you should NEVER try to do anything to a
sleeping job. Jobs put themselves to sleep for a reason, and it's
impossible for the Queue System or anyone else to know why and what that job
is waiting for. However, in the case of RDB Refresh, I can tell you exactly
why this particular job sleeps (read on for more info on this...).

Second, for troubleshooting what the problem really was and how to fix it,
what would be helpful in your case is not necessarily the error from the CBS
failure, but the error from the Custom Field sync job that failed (read
below to understand what I mean by this). Also the error from the RDB
Refresh job that failed would be helpful. These 2 errors together can help
determine what went wrong.

Third, I'll try to explain the RDB Refresh process, but here is the quick
version (below is a more lengthy text version):

1. RDB Refresh job is "chosen" by the Queue to process.
2. RDB Refresh selects some number of entities (resources, custom fields,
whatever was restored) to restore/sync.
3. RDB Refresh places one job in the Queue for each entity it chose in step
2.
4. RDB Refresh job goes to sleep (this is only an indicator to the Queue
that it should not be a candidate for processing at this time) and sets its
wakeup time for 5 minutes later.
<five minutes pass, and the entities are synced as the jobs from step 3 are
picked up for processing by the Queue>
5. RDB Refresh wakes up (this only means that its wakeup time has passed,
and the Queue will now consider it as a "processable" job).
6. RDB Refresh gets chosen for processing by the Queue System.
7. RDB Refresh assesses whether all of the jobs from step 3 have been
processed by the Queue yet.
If yes, then if there are more entities to be restored/synced, go
back to step 2.
If yes, then if there are no more entities to be restored/synced,
then go to step 8.
If no, then go to step 4.
8. RDB Refresh determines whether it should succeed or fail.
If ALL entities processed with no errors, then succeed.
Else fail.

The RDB Refresh job itself is merely a "monitor" of the admin restore
individual reporting sync jobs. It is a bottleneck of sorts, so that the
Reporting System doesn't completely flood your Queue with requests to sync
data from an admin restore all at once. Think of the organizations who have
thousands and thousands of Resources, Custom Fields, etc. So what the RDB
Refresh job does is get a certain number of entities (custom fields, in your
case) to sync, place jobs in the Queue for those entities, and go to sleep
waiting for those new jobs to get processed. Again, in order for the
Reporting System to not flood your Queue, the RDB Refresh job will sleep for
a minimum of 5 minutes in one "interval" (but could sleep longer if the
entities it is waiting for are taking longer than 5 minutes), no matter
what. If any ONE of the entities it is trying to sync fails to do so, then
the RDB Refresh job itself will fail. If ALL entities synced just fine,
then the RDB Refresh job will succeed. This is just a way to tell you if
any one of the entities failed, so you can investigate.

 

[Mike Mahoney]

Sharry

Thanks for the in depth response. For the record i ahve extracted elemennts
of the event log, displayed in reverse order (earliest event first)

The description for Event ID ( 5000 ) in Source ( Office SharePoint Server
Error Reporting ) cannot be found. The local computer may not have the
necessary registry information or message DLL files to display messages from
a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve
this description; see Help and Support for details. The following information
is part of the event: ulsexception12, w3wp.exe, 6.0.3790.1830, 42435be1,
microsoft.office.project.server, 12.0.4518.0, 454185f3, 12615, 59,
inconsistentinternalstateexception, 0000.

The description for Event ID ( 7797 ) in Source ( Office SharePoint Server )
cannot be found. The local computer may not have the necessary registry
information or message DLL files to display messages from a remote computer.
You may be able to use the /AUXSOURCE= flag to retrieve this description; see
Help and Support for details. The following information is part of the event:
PSI Entry Point:
Project User: CHEYNE\mmahoney
Correlation Id: e6e94537-81de-439f-b688-640f24113c8e
PWA Site URL: http://ccm-devprj01/PWA
SSP Name: SharedServices1
PSError: ReportingProjectChangeMessageFailed (24006), Project
UID='6fc8a0d0-0720-4169-a305-f7edf3ef1864'. PublishType='All', Invalid
project type InactiveGlobal. This type of project cannot pe transfered to
reporting database. (Repeated)


The description for Event ID ( 7704 ) in Source ( Office SharePoint Server )
cannot be found. The local computer may not have the necessary registry
information or message DLL files to display messages from a remote computer.
You may be able to use the /AUXSOURCE= flag to retrieve this description; see
Help and Support for details. The following information is part of the event:
PSI Entry Point:
Project User: CHEYNE\mmahoney
Correlation Id: e6e94537-81de-439f-b688-640f24113c8e
PWA Site URL: http://ccm-devprj01/PWA
SSP Name: SharedServices1
PSError: GeneralQueueJobFailed (26000),
8d802909-407c-4e86-951a-6bdf218cc767, CCM-DEVPRJ01, ReportingProjectPublish,
ReportProjectPublishMessageEx, 1,

The description for Event ID ( 7799 ) in Source ( Office SharePoint Server )
cannot be found. The local computer may not have the necessary registry
information or message DLL files to display messages from a remote computer.
You may be able to use the /AUXSOURCE= flag to retrieve this description; see
Help and Support for details. The following information is part of the event:
PSI Entry Point:
Project User: CHEYNE\mmahoney
Correlation Id: e6e94537-81de-439f-b688-640f24113c8e
PWA Site URL: http://ccm-devprj01/PWA
SSP Name: SharedServices1
PSError: ReportingRDBRefreshMessageFailed (24023), One of the stages of the
Refresh operation failed, RDB area: Epm, error mode: ContinueOnErrors, lock
RDB on errors: False, refresh sleep time: 00:05:00. (Repeated)

The description for Event ID ( 7704 ) in Source ( Office SharePoint Server )
cannot be found. The local computer may not have the necessary registry
information or message DLL files to display messages from a remote computer.
You may be able to use the /AUXSOURCE= flag to retrieve this description; see
Help and Support for details. The following information is part of the event:
PSI Entry Point:
Project User: CHEYNE\mmahoney
Correlation Id: e6e94537-81de-439f-b688-640f24113c8e
PWA Site URL: http://ccm-devprj01/PWA
SSP Name: SharedServices1
PSError: GeneralQueueJobFailed (26000),
be5cc7db-5446-4164-9218-70ef00a278ba, CCM-DEVPRJ01, ReportingRefresh,
ReportRefreshMessage, 1,

The description for Event ID ( 7689 ) in Source ( Office SharePoint Server )
cannot be found. The local computer may not have the necessary registry
information or message DLL files to display messages from a remote computer.
You may be able to use the /AUXSOURCE= flag to retrieve this description; see
Help and Support for details. The following information is part of the event:
PSI Entry Point:
Project User: CHEYNE\mmahoney
Correlation Id: 8becd370-5f46-4acd-9601-055ecb2a38bb
PWA Site URL: http://ccm-devprj01/PWA
SSP Name: SharedServices1
PSError: CBSMetadataProcessingFailure (17005), Setting
UID=00007829-4392-48b3-b533-5a5a4797e3c9 ASServerName=ccm-devprj01
ASDBName=PWADev ASExtraNetAddress=ccm-devprj01 RangeChoice=0 PastNum=1
PastUnit=0 NextNum=1 NextUnit=0 FromDate=02/19/2007 13:07:44
ToDate=02/19/2007 13:07:44 HighPriority=True, Invalid call to
GetJoinForCustomFieldDimension. Cannot join "Task Non Timephased" fact with a
custom field of entity Assignment

Thanks

Mike

 

[Sharry Heberer [MSFT]]

How many custom fields do you have which would have been restored? I
believe the error that caused RDB Refresh to fail would have been quite a
bit (in time) before the RDB Refresh failure. At least 5 minutes before, if
you have very few custom fields. It could be hours before if you have very
many custom fields. What would help the most is if you look at your Manage
Queue page, filter the Job Types to only RDB Refresh, Reporting Custom Field
Sync and Reporting Lookup Table Sync. Don't forget to change the History
filter to be whatever day(s) it was that you did the Restore. If you did
this less than 1 week ago, and there were any failures, then they should
show up here. Unfortunately, if the failure was more than a week ago, then
the jobs have already been cleaned up and removed from the Queue. (by the
way - you probably won't see the jobs that succeeded, because successful
jobs get cleaned up after 24 hours, so don't worry about that) If you still
don't see any failures for those types of jobs I specified (the error will
be linked to from the "Error" column in the Jobs grid), then I don't see
what the problem can be and I can take this down for investigation and
consideration to be fixed in a future release.

Please let me know how it goes....
Sharry

--
This posting is provided "AS IS" with no warranties, and confers no rights.

Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

 

[mike.mahoney]

How many custom fields do you have which would have been restored? I
believe the error that caused RDB Refresh to fail would have been quite a
bit (in time) before the RDB Refresh failure. At least 5 minutes before,if
you have very few custom fields. It could be hours before if you have very
many custom fields. What would help the most is if you look at your Manage
Queue page, filter the Job Types to only RDB Refresh, Reporting Custom Field
Sync and Reporting Lookup Table Sync. Don't forget to change the History
filter to be whatever day(s) it was that you did the Restore. If you did
this less than 1 week ago, and there were any failures, then they should
show up here. Unfortunately, if the failure was more than a week ago, then
the jobs have already been cleaned up and removed from the Queue. (by the
way - you probably won't see the jobs that succeeded, because successful
jobs get cleaned up after 24 hours, so don't worry about that) If you still
don't see any failures for those types of jobs I specified (the error will
be linked to from the "Error" column in the Jobs grid), then I don't see
what the problem can be and I can take this down for investigation and
consideration to be fixed in a future release.

Please let me know how it goes....
Sharry

--
This posting is provided "AS IS" with no warranties, and confers no rights.

Use of included script samples are subject to the terms specified athttp://www.microsoft.com/info/cpyright.htm













...

read more »- Hide quoted text -

- Show quoted text -

Sharry

Only two entries come up for these event categories, both essentially
the same. Does not look too helpful but pasted here:
Error summary/areas:
Reporting message processor failed
ReportingRDBRefreshMessageFailed
ReportingRDBRefreshMessageFailed
ReportingRDBRefreshMessageFailed
ReportingRDBRefreshMessageFailed
ReportingRDBRefreshMessageFailed
ReportingRDBRefreshMessageFailed
Queue
GeneralQueueJobFailed
Error details:

<?xml version="1.0" encoding="utf-16"?>
<errinfo>
<general>
<class name="Reporting message processor failed">
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="76e508bf-a246-4412-b783-439ada54359a" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="b3a44ab6-5b82-433d-9a9b-34c9f60e97d5" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="174186d1-58bf-4484-a62d-7759007a1810" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="1d6aec11-c91f-45ad-9cde-68ba33b6aab5" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="8be3f283-d9ca-4c96-b257-db1ae8fe2315" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="0fbed41c-3696-4b0f-924d-e233bba50fac" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
</class>
<class name="Queue">
<error id="26000" name="GeneralQueueJobFailed"
uid="f2cc17f2-57d4-425f-a3b5-65ce0405babd"
JobUID="aeda5a2f-5b1b-440d-8a8e-98f9fd213cf3" ComputerName="CCM-
DEVPRJ01" GroupType="ReportingRefresh"
MessageType="ReportRefreshMessage" MessageId="1" Stage="" />
</class>
</general>
</errinfo>

 

[Sharry Heberer [MSFT]]

So nothing for Custom Fields or Lookup Tables, eh (those errors you sent are
for RDB Refresh messages, not CFs or LTs)? OK, I will investigate on my
end. Thanks for the info!

--
This posting is provided "AS IS" with no warranties, and confers no rights.

Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

How many custom fields do you have which would have been restored? I
believe the error that caused RDB Refresh to fail would have been quite a
bit (in time) before the RDB Refresh failure. At least 5 minutes before,
if
you have very few custom fields. It could be hours before if you have
very
many custom fields. What would help the most is if you look at your
Manage
Queue page, filter the Job Types to only RDB Refresh, Reporting Custom
Field
Sync and Reporting Lookup Table Sync. Don't forget to change the History
filter to be whatever day(s) it was that you did the Restore. If you did
this less than 1 week ago, and there were any failures, then they should
show up here. Unfortunately, if the failure was more than a week ago,
then
the jobs have already been cleaned up and removed from the Queue. (by the
way - you probably won't see the jobs that succeeded, because successful
jobs get cleaned up after 24 hours, so don't worry about that) If you
still
don't see any failures for those types of jobs I specified (the error will
be linked to from the "Error" column in the Jobs grid), then I don't see
what the problem can be and I can take this down for investigation and
consideration to be fixed in a future release.

Please let me know how it goes....
Sharry

--
This posting is provided "AS IS" with no warranties, and confers no
rights.

Use of included script samples are subject to the terms specified
athttp://www.microsoft.com/info/cpyright.htm













...

read more »- Hide quoted text -

- Show quoted text -

Sharry

Only two entries come up for these event categories, both essentially
the same. Does not look too helpful but pasted here:
Error summary/areas:
Reporting message processor failed
ReportingRDBRefreshMessageFailed
ReportingRDBRefreshMessageFailed
ReportingRDBRefreshMessageFailed
ReportingRDBRefreshMessageFailed
ReportingRDBRefreshMessageFailed
ReportingRDBRefreshMessageFailed
Queue
GeneralQueueJobFailed
Error details:

<?xml version="1.0" encoding="utf-16"?>
<errinfo>
<general>
<class name="Reporting message processor failed">
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="76e508bf-a246-4412-b783-439ada54359a" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="b3a44ab6-5b82-433d-9a9b-34c9f60e97d5" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="174186d1-58bf-4484-a62d-7759007a1810" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="1d6aec11-c91f-45ad-9cde-68ba33b6aab5" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="8be3f283-d9ca-4c96-b257-db1ae8fe2315" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
<error id="24023" name="ReportingRDBRefreshMessageFailed"
uid="0fbed41c-3696-4b0f-924d-e233bba50fac" QueueMessageBody="One of
the stages of the Refresh operation failed" Error="RDB area: Epm,
error mode: ContinueOnErrors, lock RDB on errors: False, refresh sleep
time: 00:05:00" />
</class>
<class name="Queue">
<error id="26000" name="GeneralQueueJobFailed"
uid="f2cc17f2-57d4-425f-a3b5-65ce0405babd"
JobUID="aeda5a2f-5b1b-440d-8a8e-98f9fd213cf3" ComputerName="CCM-
DEVPRJ01" GroupType="ReportingRefresh"
MessageType="ReportRefreshMessage" MessageId="1" Stage="" />
</class>
</general>
</errinfo>