[Hennepin]

We recently upgraded from Accounting for Nonprofits to Financial Edge and now find that generating a report is slower than grandma trying to navigate an icey Minnesota sidewalk with her walker. Reports that used to take a few seconds to a few minutes under ANP now take ten minutes to several hours with FE. Alarm bells have been sent up our sales rep food chain because the application is largely unuseable as it stands, but I thought I would check here to see if anyone else has encounter this issue and figured out a solution.

Our server CPU is near the low end of recommended processors for the application. RAM is healthy at 2 gig for the server. Workstations are mostly new with a gig of RAM under the hood. When someone kicks off a report I can watch the server CPU spike up to 100% and stay there until the report is done. Other users say they can continue working fine while this is going on, which I find strange. I figure they would also be bogged down while the server is being hammered, but their testimonials are anecdotal so I take them with a grain of salt.

At the moment I'm researching new servers, but I would hate to go through the time and heartache of a swap if I don't have to. Plus the users would get out the tar and pitchforks if the swap only reduced the report time from 2 hours to 1 hour 50 minutes. Bad Things would happen when the townspeople storm the castle.

Thanks,
-Todd

[JoeSullivan]

Todd,
You are not alone. We have had similar speed issues with reports and are very familiar with tar and pitchforks. The speed seems to be greatly effected by how much summarizing needs to be done in the report. I have not had the patience yet to sit down and figure out what specific configuration in the report cause the slowdowns but I have been told by Blackbaud.. "oh, yeah. it's because...(I wish I could remember what they said here but trust me it was very vague)" only to go to my Finance department and have them say that this was the main thing they needed to report on.

My solution was just to create my own Crystal Reports that pull the data directly from the data tables in FE. I do not recommend creating a custom report in FE with the same data because it will probably take just as long. I believe I was able to load up a report that took FE 30min to pull up in like 10 seconds. If you don't have someone in your organization that knows the FE data well and can create views in your SQL server this might not be a good option for you. I really hope they improve their reporting in the future because we have experience the same 2-4hr process for some reports.

If this helps you on whether or not upgrading your severs will help, we currently are running a dual core 2.4ghz Xeon processor with 4gb of ram. I believe our drives are 70gb striped but I'm not 100% sure without checking. We are upgrading our severs very soon and I can let you know if we see any improvement in our reporting. We have focused mainly on the drives we select since we have notice a large amount of disk activity when running reports. Sometimes reading the disk can be the biggest bottleneck.

Good luck with the townspeople. I know it sucks being the one everyone blames when an application you have nothing to do with is slow.. then trying to convince them that the software is great and that the upgrade was a good thing. I feel your pain.

Joe Sullivan
Database Admin/Programmer
Great Commission Ministries

[Hennepin]

Joe,

Thanks for the input--it gives me good food for thought. Right now I've got our database in with Blackbaud so they can run it on a few test boxes to see how long the reports take. One of the VPs hit me up in the bathroom this morning and wanted me to just order a new box and call it done. I had to patiently explain that it's not a cheap purchase and we want to make sure we're getting the right hardware. His head didn't quite pop off when I explained that we're looking at a $4000 - 6000 purchase, but it did look like he wanted to do a Linda Blair on me. Maybe he thought this would be a $600 IDE workstation purchase.

It sounds like your current server is similar to what I'm buying. I'm looking at a 3 GHz CPU, 1333 MHz FSB, 2 gig RAM, and 15k SCSI drives with a raid5. But if you already have one like that in your back pocket and it's still not making honey for you, I don't want to spend the money to find I'm right back where I started. That's like getting a flat and you simply rotate the tired and drive home.

Outside of reports, do you also have any issues with normal processing? We find that when someone posts that it tends to drag the other users down. Shoot, I don't even use the application and it's dragging me down.

The users have been pretty good with me so far. They realize it's a Blackbaud issue and that I'm in their corner and here to solve the issue. Don't shoot the guy that's here to help you or you may end up with a screen shot of your desktop as your wallpaper and all of your icons hidden.

-Todd

[Darlene]

Todd- (and Joe)-

I am not technically inclined, so much of what the two you are discussing is beyond me. I am however, fairly familiar with what FE likes and doesn't like. I'm curious as to the type of report that is causing so much pain. My personal experience has been with GL reports if you filter by projects you could retire before you see the report. A much quicker report is to filter by department and format the sort by project. There maybe some other tools I could share with your users
PS -Joe we really love you!

[Hennepin]

Darlene,

So far all functions have been a doggin' it with regards to FE, not just reports. Certainly reports that used to take seconds now take minutes and those that took minutes now run for hours. But other tasks such as posting are as reluctant as a dog who doesn't want to go outside to do his business on a cold winter day.

I'm still waiting to get testing results back from Blackbaud, but Joe's experience with disk useage doesn't give me that warm fuzzy feeling. Hopefully he;ll spot this thread again and let me know what the RPM speed on his drives are. If they're 10k, then I've got a little wiggle room. But if he's already usinig 15k drives, I might as well put FE out to pasture and start looking for a new pony to ride.

-Todd

[Hennepin]

Update:

The new server came in this week and I spent a couple of days setting it up, so now we're on to testing. The server stats are what I posted above:

3.0 GHz CPU with four threads
2 gig ram with 1133 MHz FSB
15k SCSI drives on raid5

Here are the test results with running the same report each time.

Old server: 2 ½+ hours
Blackbaud test server: 1 hour
New server: 22 minutes
New server with some years summarized: 22 minutes
New server with years purged: 20 minutes

That's my data and I'm sticking to it.

-Todd

[ajibril]

Hennepin,

How much does this system you have posted cost?

Thanks,

[Hennepin]

It's a Dell 2950 and the cost was...$4,674.00.

-Todd

[TFA_Glen]

Hi, just thought I'd add my two cents... we recently upgraded from 7.6 to 7.70 and our Income Statement reports went from 10 minutes to 1 hour.

Of course, FE support testing can't confirm our experience. Their 7.6 test box shows 40 minutes not 10. (Which is still unacceptable)

FE server is new: 2 dual core procs, 4gb ram.

This is what BB support wrote to us:
"I believe I have narrowed down the cause to the project filters being applied to the columns. I did some further testing on income statement "ERF source data - nps." Four of the columns had projects filtered by queries. I removed these filters and the report processed in under four minutes.
Since projects are assigned to individual transactions, reports filtered by projects must consider each individual transaction to process. Therefore, in The Financial Edge 7.70, we created Project Financial Statements, specifically designed to increase speed for running financial statements with projects. I suggest to test running this report to see if this will meet your reporting needs. This report can be found in The General Ledger / Reports / Project Reports / Project Financial Statements. Please let me know if this helps."

Suffice to say, his solution didn't work. It's a little frustrating since we had no problems with 7.6. Incidentally, I think prior to 7.7 you could backdoor odbc into FE, but they seem to have shut that down.

I like Joe's method of sucking data into a sql solution but we don't have the advanced support licensing so we can't get at the tables in the database.

Anybody else out there having issues with Income Statement reports that utilize project subqueries?

Glen

[Hennepin]

Glen,

What does your server have for hard drives as far as speed goes? I assume they're SCSI class and not IDE. I brought that up because even though my gut reaction is we're dealing with poor coding, your hard drives are going to be the slowest subsystem on the server. If they're not up to snuff they'll slow things down quite a bit, even if the software has decent code.

My guess is that the report generating functions have become a secondary consideration for BlackBaud. I don;t have any inside information, but it wouldn't surprise me if they work hard to get a new version and enhancements out and the report queries aspect of the code doesn't get the attention it needs.

Besides complaining, there's not a lot we can do to prompt BlackBaud to devote more resources to reports. Which is why I'm looking at hardware to see if we can squeeze a little more speed out of the box. I also like Joe's method of using Crystal Reports to run queries, although not everyone is going to have the budget or in house expertese to go that route. And it strikes me as funny and silly that we have to scrap FE's entire reporting module in order to get the job done.

-Todd

[TFA_Glen]

Todd
Thanks for your reply. I checked and we do use SCSI. Our hardware is definitely up to snuff. I don't think it's an HD issue tho b/c our income statement reports ran in 10 min prior to the last upgrade. something happened with the last upgrade. We rely on FE's income statement reports to drive our budgeting, forecasting, and modeling (that's how we tie actuals to budget/forecast). It's behavior like this that's driving me to engage non BlackBaud p&a vendors because I want to shave the planning process off of FE, which apparently works best as a transactional db for accounting, and not much else. If I can sync an rdb/olap to FE to siphon off the transactions (say nightly), I can say adios to FE for our reporting/querying/forecast versioning needs.

So i'm basically seconding your statement "...it strikes me as funny and silly that we have to scrap FE's entire reporting module in order to get the job done." and exploring something along the lines of Joe's crystal solution by attaching yet another sql server (with an application designed specifically for P&A on top of that) to facilitate our budgeting process.

Best,
Glen

[JoeSullivan]

I am currently looking at a data warehousing solution that would universally solve all of our reporting needs. If all goes well I'm thinking of ditching all Blackbaud reporting and moving towards MS Analysis/Reporting Services (where Blackbaud is headed anyways). I'm sure if we had the financial resources to invest in TIE that would probably help but we just can't afford it.

I know not everyone has the resources in-house to put together a DW solution, but if you do that could work for you. Sounds to me like you might already be headed that direction anyways. I would love to provide any expertise that I find, although I'm still learning myself.

If I understood correctly at the conference, I believe the Infinity platform will come with some data warehousing capabilities and I'm hoping much better reporting as well, but we just can't wait for it. I guess what I'm implying is that Infinity will probably solve all of these problems. I know that doesn't help the here and now but we can at least have hope for the future of reporting in Blackbaud.