[Nfbf-l] DBS report
RJ Sandefur
joltingjacksandefur at gmail.com
Wed Jun 27 22:24:41 UTC 2012
For those who did not read the report, here it is!
a COMMISSION'S BRIEFING
PACKAGE
DIVISION OF BLIND SERVICES
MANAGEMENT REVIEW
PREPARED BY THE DOE
OFFICE OF INSPECTOR GENERAL
MAY 7, 2012
Commission's Briefing Package
Division of Blind Services
Management Review
Briefing Package
CONTENTS
1) DBS Management Review - High Level Issues
2) Climate Survey
a. Survey instrument used
b. Selected survey results (graphs)
c. Sample responses to climate survey
d. DVR climate survey - final report (excerpt)
3) Staff interviews (selected sample)
4) Contracting
a. Review results
b. DFS preliminary findings from recent contract
audit
c. OIG 2008 audit report entitled "Contracted and
Purchased Client Services"
d. Additional issues for OIG audit/ investigation
5) Personnel actions review
DBS Management Review - High Level Issues
What we did:
Developed and administered an on-line anonymous employee climate survey. The
climate survey contained 40+ opinion statements (questions) regarding job satisfaction,
communications, supervision, and management practices, including contracting
practices. The survey resulted in an 83% (224/270) response rate, with roughly onethird of respondents providing supplemental written comments.
Approximately 45 current and former employees were interviewed either in person or
telephonically (about half were HQ staff and half were field office staff). We also
interviewed Florida Department of Financial Services (DFS) auditors who recently
performed an audit of DBS contracts. We reviewed procedures and practices for DBS
client services contracts. Finally, we reviewed DBS human resources records
supporting personnel actions.
Working Environment Roughly speaking, 40% of DBS staff interviewed support
Division management, 40% do not (often passionately), and 20% are on the sidelines.
Compared to results of a recent Florida Division of Vocational Rehabilitation climate
survey, DBS climate survey results were much more negative. DBS climate survey
comments expounded on issues such as:
·
·
·
·
·
·
·
Too much micro-management from senior management
Punitive, not positive atmosphere
Many DBS staff fear making a decision/statement that is not favorably received
retaliation likely
More training needed
Fear trickles down to all employee levels
Dissatisfaction over pay DVR gave raises to counselors because the director
was willing to fight for raises.
DBS has become more accountable with more written policies/procedures.
Communications
· Better communication needed from the top down.
· Executive management should communicate more with field supervisors and
employees (not just managers).
· Continually changing policies and procedures confuses employees.
Staff Turnover
· Employee turnover has been perceived as rampant.
· Jobs not filled but simply assumed by those employees remaining.
· Staff fear for their jobs daily and try to keep off the radar.
·
Some view changes as necessary and positive.
Hiring Process
· Education and experience should be dominant factors in the hiring process.
· Not all vacancies are made public/advertised on People First.
· Employees hired/promoted to manage others need the correct skills to do the
job.
· Concerns about staff pay equity.
Client Services Contracts The intent of chapter 413, F.S. may be to provide the
maximum amount of funding to the community rehabilitation providers (CRPs),
however, the Division could be going too far in practice. CRP funding has increased
and services provided by DBS employees have decreased.
DFS found that contract performance was not aligned with payments, financial remedies
are assessed after the full contract payment (when it may be more difficult to collect),
and monitoring is not adequate vendor reports are just accepted and not verified.
Overall, the DFS auditors commented that the division is too "cozy" with contractors.
·
·
·
·
Lack of competition with the same 15-20 providers used each year.
Fixed rate contracts with monthly payments. Liberal terms for payment.
Contract terms not always enforced.
Lack of effective negotiation of contracts. Lack of monitoring. Lack of close out
reports.
CRPs are favored when other alternatives may provide better client service; in
effect, CPRs have a monopoly.
Conclusion We identified a number of indicators that change needs to take place in
DBS. Contract administration should be strengthened, communications improved,
personnel management should be improved and made more transparent. There is a
need to reach out more to help make staff feel that they are included. We have
identified a number of issues from interviews with DBS current and former staff that are
either under current investigation or are subject to future investigation.
Greg White
May 4, 2012
Division of Blind Services Employee Climate Survey
Thank you for participating in the Division of Blind Services Employee Climate Survey conducted by the
Department of Education's Office of Inspector General. The purpose of the survey is to provide feedback to
Department of Education management regarding your workplace environment.
Responses are anonymous and computer addresses will not be stored by the survey administrator.
The survey is approximately 50 statements about your work environment and should take about 15 minutes to
complete. The survey will be available until March 8, 2012.
Rate each statement using Strongly Agree, Agree, Disagree, Strongly Disagree or No Opinion.
If you have any questions about this survey or need assistance, please contact us by phone at (850) 245-0403 or
by email at oig at fldoe.org.
Definitions:
DBS stands for the Division of Blind Services.
Management refers to the division director, deputy directors, bureau chiefs and district directors.
Office refers to the office or district in which you work.
Supervisor refers to the person who approves your timesheet.
*All statements require a response. There is a "No Opinion" option for each.
Job Satisfaction:
1. *I am satisfied with my involvement in decisions that affect my work.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
2.*Work is assigned fairly in my office.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
3. *My workload allows me to do my job effectively.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
4. *I receive a fair evaluation of my job performance.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
5. *I have the information and training I need to effectively do my job.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
6. *Overall, I have positive working relationships with employees in my office.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
7. *I feel physically safe and secure at work.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
8. *Overall, I am satisfied with DBS as a place to work.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
9. *I am proud to say that I work for DBS.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
10. Please provide suggestions in the input box below that you think would improve your office or district.
(Optional)
Communications:
11. *There is a way for me to discuss my concerns with my management.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
12.*I am kept informed about matters that affect my job.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
13.*My office staff communicates with each other as needed.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
14.*Communication from management is clear.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
15.*Employees are free to speak up and say what they think.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
16.* I rarely receive conflicting instructions from different levels of management on how to do my job.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
17.* My office has effective communications.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
18. Please provide suggestions in the input box below you think would improve communications. (Optional)
Supervision:
19. *My supervisor is responsive to my needs and concerns.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
20.*My supervisor promotes a work environment that fosters productivity.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
21.*My supervisor treats me with dignity and respect.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
22.*My supervisor frequently acts as a positive role model for employees to follow.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
23.*My supervisor is supportive of staff and provides assistance as needed.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
24.*My supervisor periodically provides constructive feedback on my job performance.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
25.*If I have a conflict with my supervisor, I can take it to the next level without fear of retaliation.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
26.*My supervisor is effective in his/her job.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
27. Please provide suggestions in the input box below that you think would improve supervision. (Optional)
Management Practices:
28. *My management provides effective leadership.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
29.*Ethical behavior is promoted by management.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
30.*I understand the goals and objectives of my office.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
31.*The policies and procedures provided to me are sufficient to help me do the best job I can.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
32.*Contracting for services with outside entities is conducted in a cost effective manner.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
33.*Contracts are adequately monitored to ensure deliverables are provided.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
34.*Contractors provide high quality services to clients.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
35.*My office's positions are generally filled in a timely manner.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
36.*Positions are filled with qualified people.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
37.*The hiring process for vacancies is conducted fairly.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
38.*DBS employees who perform well are provided opportunities for advancement.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
39.*I am not aware of discrimination (race, gender, or age) in my office.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
40.*I am not subjected to inappropriate conversations that may cause me to feel uncomfortable.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
41.*Management practices in my office promote organizational success.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No Opinion
42. Please provide suggestions in the input box below that you think would improve management practices.
(Optional)
General Information:
43. If there are any other comments that you would like to share or issues that you would like to raise,
please do so here: (Optional)
44. I have been employed at DBS for: (Optional)
Less than 1 year
1-2 years
2-5 years
5-10 years
More than 10 years
45. I am a: (Optional)
Staff Member
Supervisor
Manager
Other
46. I am located in: (Optional)
Tallahassee Headquarters
District
Other
Your feedback is important. Thank you for participating!
DBS Climate Survey - selected graph results
224 responses
Job Satisfaction
Statement 1: I am satisfied with my
involvement in decisions that affect my work.
Job Satisfaction
Statement 4: I receive a fair evaluation of my
job performance.
Job Satisfaction
Statement 8: Overall, I am satisfied with
DBS as a place to work.
100%
75%
50%
25%
0%
49%
33%
2%
17%
100%
75%
50%
25%
0%
37%
13%
4%
46%
100%
75%
50%
25%
0%
55%
12%
30%
3%
Communications
Statement 14: Communication from
management is clear.
Communications
Statement 15: Employees are free to speak
up and say what they think.
Supervision
Statement 19: My supervisor is responsive
to my needs and concerns.
100%
75%
50%
25%
0%
41%
31%
3%
25%
100%
75%
50%
25%
0%
40%
19%
3%
38%
100%
75%
50%
25%
0%
65%
19%
3%
13%
"Agree" and "Strongly Agree" responses were consolidated to "Agree" for reporting purposes. The same for "Disagree" and "Strongly Disagree".
Page 1 of 2
DBS Climate Survey - selected graph results
224 responses
Supervision
Statement 25: If I have a conflict with my
supervisor, I can take it to the next level
without fear of retaliation.
100%
75%
50%
25%
0%
Management Practices
Statement 28: My management provides
effective leadership.
Management Practices
Statement 32: Contracting for services with
outside entities is conducted in a cost
effective manner.
100%
75%
50%
25%
0%
47%
26%
8%
19%
100%
75%
50%
25%
0%
47%
35%
4%
14%
26%
26%
17%
30%
Management Practices
Statement 33: Contracts are adequately
monitored to ensure deliverables are
provided.
100%
75%
50%
25%
0%
Management Practices
Statement 36: Positions are filled with
qualified people.
Management Practices
Statement 37: The hiring process for
vacancies is conducted fairly.
38%
26%
17%
20%
100%
75%
50%
25%
0%
43%
30%
8%
19%
100%
75%
50%
25%
0%
38%
16%
30%
16%
"Agree" and "Strongly Agree" responses were consolidated to "Agree" for reporting purposes. The same for "Disagree" and "Strongly Disagree".
Page 2 of 2
Responses to DBS Employee Climate Survey
Question 43. If there are any other comments that you would like to share or issues
that you would like to raise, please do so here:
Note: First 50 responses are included here (from total responses of 77)
1. State office verbalizes open door policy for communication; but sometimes I
wonder about it in practice.
2. We have not had a raise in over 6 years, but Division of Voc. Rehab has and
they are in the same DOE
3. Work hours are to be kept for that and not to have any additional work after hours
or to take home without you being able to disagree with the request from your
superior.
4. I feel when you do a good job and go over and above, that should count on your
evaluations.
5. At the facility where I work, a person was promoted who did not have as much
experience as others with similar skills. This person had not had real leadership
experience as well. She was known through her family for years to the person
who wished to hire her. The person doing the hiring also had told someone who
should have been a candidate, that he wanted someone who would be around
longer. This to me is age discrimination.
6. It seems like a few years back I heard that there will be less paperwork involved
and more electronic options. The electronic options are there, but more
paperwork is being generated and files are becoming larger and unmanageable.
7. I have spent many years with this agency and until the last couple of years was
extremely proud of the work we did, the work I did, and always felt very gratified
in my job. I felt I was a part of the agency and even when I disagreed with
management's decisions, I could voice that, without fear. In the last few years I
have feared that I would be fired every day I come to work. I meet and exceed
every deadline given, my work quality is extremely good, I follow directions, and
have always had a strong work ethic.
To have my job threatened because one person at a CRP thinks I'm too stern, or
due to a personality clash is unfair, but has happened. I have had my supervisor
call me into a meeting with other staff to recite a list of things those in the room
have heard about me, an inquisition of sorts. It was the most humiliating
experience of my life. During this `meeting' I asked for facts, none could be
provided. In the end it was determined that the accusations were unfounded
because staff admitted they were just `assuming' things with no real fact. The
supervisor should have done her homework before hand, but instead I was put
on the chopping block. At the end of the meeting the supervisor stated that all
was settled, told those that had brought this information to her that in the future
they need to have facts not hearsay or rumor and wanted to `hug' everyone.
Over the years, I have always excelled in my job, moved ahead, had outstanding
evaluations, however in the last few years my professionalism has been
questioned, my work, and more and I find it extremely offensive. I have done
everything asked of me and yet it never seems to be enough for management. It
is my feeling that once management decides a person is a `problem' nothing will
change that view. It also seems that a person is only considered a `problem'
when they question a process, when they question anything indicating it is not in
line with agency policy, or regulations. The environment for many (probably more
supervisor and management positions) is thick with fear.
We have seen more turn over in the last 2 years then we have seen in 10, 20 or
more. There have been several re-orgs that move staff into positions they may
not want but have no choice in as it's that or termination. It's been a very
unhealthy working environment and a confusing one, as it does seem we are
being dismantled and the CRP's are taking over our jobs, controlling our money
and client's are no longer given a choice when it comes to services. They are told
we contract with CRP's and if they want services they get them through the CRP
or nothing. We don't put bids out for services, so no other provider is allowed to
work with our client's, only our current CRP's. Money often seems to be given to
a few certain CRP's while it continues to be cut from our client services budgets
reducing what we are able to do for our clients.
It's just become a very frustrating, stressful, fearful place to work, which is very
sad. Until these last few years, I loved my job and wouldn't hesitate to tell people
how much I loved it, how I never wanted to leave it and really encourage folks to
come to DBS. Unfortunately I can't say that any longer. I feel deflated,
unnecessary, and unappreciated. I would love to feel excited and really good
about DBS again!!
8. It feels like we are always working in fear of losing our jobs and not because we
did something wrong.
9. In our workplace there is discrimination, favoritism, and harassment going on but
no one wants to speak up because we all need our jobs. I have seen employees
leave crying because they were treated so badly they felt forced to resign or
forced to go to EAP...I wish some changes could be made to make this a positive
workplace. I think sometimes power takes the place of what our mission really is
here...which is the customer.
10. We are no longer a 'team'. Segregation (not in race, but in professional status) is
the number one rule. If you are not `a Senior Professional Team Member' you are
no-one that matters. Knowledge no longer matters, it's who you know and who's
friend it is for promotional opportunities or hiring. I have worked at this agency
over 15 years and have received many Letters of Accommodation and Awards
for my work and abilities; but because I helped someone else who was not in my
`unit' but was a Blind Services employee here at headquarters, I was written up
and told not to help anyone again unless the `Senior Management Team'
approved it.
Now I feel that if I make one of `their kind' of mistakes I will be fired. I come to
work each day not knowing if I'm next in line to be let go or who will it be today.
When they forced a co-worker to resign, I was called into that office and was told
to my face that I would never survive. A few months passed and I was called into
the office again and was asked when I was going to retire? I am 58 years old and
a widow and they want me to retire. They told me I could sell my crafts that I
make in order to survive. I am unable to do so. I now know that it won't be long
before they build a case on me in order to force me to resign as well. I have
received all fours and fives on all my evaluations for years accept for this last one
where I received a 3.5; that was the lowest score I have ever received on any of
my evaluations. We have lost many good employees because they have been
forced to resign; then you actually hear them say well, I've fired 18 people this
year and I don't think I'm finished yet. People ask why you're paranoid! I need my
job no matter.
11. In hiring new Staff at any level, management or clerical, there is a constant
pattern first, most of the new hires- are not qualified, get higher salaries, and staff
that is internally qualified get demotions, or forced to downgrade their job
descriptions, changing their job titles, in part due to their own buddy system.
12. I am willing to be contacted to verbally discuss my responses.
Big differences in local office team and management.
.
13. I think as part of the evaluations for supervisors and management, there should
be a section that is rated by those they immediately supervise and/or manage.
This would allow for a capturing of employee's opinions on the effectiveness of
their leadership.
14. At one time I could see myself working for DBS 30 years and retiring, not
anymore that has changed the last couple of years. I like my job, I love what I do;
but the management has made my job very challenging.
15. The first and foremost responsibility of authorities is the well being welfare over
those to whom you have authority. The State of Florida Department of Education
has failed in that responsibility. If the State's idea is to drive away good
employees, ruin morale, and downgrade the value of workers, then it has
accomplished much. As part of the DOE, DBS is an inappropriate fit, and it's
employees cannot be evaluated by a broken gauge of expectations and
standards of teachers.
The recent downgrading of employee evaluation levels by forcing them into a
curve of mediocrity only adds an additional sword to the arsenal of nonmotivational weaponry aimed at workers. To be told that the downgrade was a
good and respectable grade is insulting. Nobody congratulates someone on their
mediocre accomplishment of a solid C! To then be told we should avoid any
negativity in the workplace, and constantly reinforce that we are sacrificing for the
good of the many, and that money isn't everything, makes one wonder where we
are really working.
The State DOE hires at below market salaries sending a poor message that
public servants are less valuable. To offer hollow hopes of future opportunities,
provide no raises for six years, no bonuses; not even a cost of living increase,
raise the cost of employee benefit premiums, cut the pension in half, and force
employees to contribute without even an opt out is shameful and dishonorable.
Employees who stay have no morale and just go through the motions as the
DOE doesn't reward good work.
Morale is non-existent. Long term employees are biding their time just waiting to
retire. Newer employees are biding their time till a better job comes along. Even
a dog is given a treat occasionally just for being good. The State DOE's
behavior toward its employees only succeeds in nailing the coffin of disrespect
and distrust that many feel for their employer. Respect given is respect earned.
If the State DOE really values its employees --- Prove it! DOEDBS --- you have a
failing grade. What a sad way to be.
16. My District Administrator is ethical, effective, empathetic and professional. My D
A deserves high marks and high praise. My supervisor, on the other hand, plays
favorites, is often heavy-handed and disrespectful to staff, and is neither
effective, ethical or professional. My supervisor does not have the training,
experience, judgment, ethics or temperament to be effective in this position.
17. I have been employed 35 years and some of my co-workers who have been
employed for about 5 years, jobs were ungraded but not mine, I think that was
discrimination.
18. I have ethical concerns about what I have been asked to do as it relates to
evaluating staff.
19. Most of the inadequacies noted are directly related to the management style and
practices of the director. Most employees fear for their jobs if they do not go
along with what the director wants or what certain agents of the director want
such as a few staff that are held very close to the director.
20. My Supervisor always promotes the best interest of the Division and our clients.
She is always provides clear direction to staff and is always available to provide
clarification of further details as needed. She is a strong role model for District
employees.
21. We all know that we need raises, although we are very thankful for our jobs and
insurance premiums that remain reasonable. However, we also know that there
is no money for raises at this time. As soon as there is, that would be a huge
morale booster. However, it would be nice to reward employees who do their
jobs well in other ways. This is done in our office with positive reinforcement, but
I wish there was a way to do this within the agency. Many feel that those who do
not do their jobs well get the same congrats as those who do. That is very
frustrating.
22. DBS needs to consider cost sharing with clients so they have enough money to
compensate the employees appropriately. Employees have lost so much in
changes to pension that without raises there is not a lot of incentive to stay
around once the economy gets better for new employees.
23. I hope this is truly confidential, because at this time I have no confidence in DOE
and its ability improve or identify the issues that have embraced this organization.
There is a climate of distrust and confidence in our mission. Real or not I feel a
toxic atmosphere, in that I sense my peers are more concern about their backs
than executing a professional program of rehabilitation.
24. DVR counselors received a raise a year or two ago while DBS VR counselors did
not. The jobs have the same duties and educational requirements. Several DBS
state office people received large raises in 2011. Some individuals that do not
meet minimum education requirements have been hired in above minimum of the
range, while others with years of experience that meet educational requirements
are started at minimum.
25. Unfortunately, in the past few years, an atmosphere of uncertainty about our jobs
has arisen as a result of some of the changes that have been seen. Vast
amounts of funding has gone to the Community Rehabilitation Programs (CRP)
for service provision, in the form of contracts, that had been paid previously, on a
cost for service basis, within which the client and VR counselor had some input.
It is now mandated that services go through the CRP, at the expense of client
choice. These include virtually all VR services with the exception of determining
eligibility, creating Individual Plans for Employment and some counseling. It is my
opinion that if RSA did not require that VR counselors do these tasks, they would
also be given to the CRP and our jobs would be considered irrelevant. I would
actually be more comfortable discussing this as opposed to writing about it, if you
wish. My cell is
26. The more I think about what I've already said it's not going to matter anyway!
27. I very happy to work for the Division of Blind Services
28. I have a concern with the mold which exists in our building, is there someone
responsible for us being checked and treated for any related issues, or is this
something that we as employees are responsible for?
29. My district office works well as a team and we work very hard to carry out the
agency mission. I believe that our current leadership has made some decisions
that have made our job a lot harder. It is unclear if these decisions were made on
lack of knowledge of what the people lower than them do. For a few years now
people from outside the agency were hired in the District Administrator position.
Some have done a good job and some (mainly due to lack of knowledge or the
rehab process and DBS) have struggled. Several would probably be much better
in a different role in the agency. Also our state office has been growing steadily
over the last few years. Yet the counselors have not had a raise or even a bonus
in 5 years. Please make sure these are anonymous because it will not be good
on anyone who said anything that is not so good.
30. The work climate in my office and throughout many offices in this agency is very
poor. People are afraid to speak up and contribute for rear of losing their jobs.
Many will not complete this survey, because they are afraid that it is not truly
anonymous and that they will be punished if they say anything negative. All of
this fear and negativity has taken place with our current director. She has fired
good people just because they fought for what is right. Also, she eliminates
people with direct blindness experience from taking supervisory positions in the
field because she does not want such people looking at contracts and service
provision. She does not want any contractors questioned about the quality of
services they provide.
31. I believe the director was sent as the hatchet person. So now we will see, again,
what you surveyors do.
32. More training in practical aspects of case management; such as effective time
management, case recordings and documentation and effective decision making
would be positive for our agency in providing better services to our clients in a
more timely and successful manner.
33. It would be helpful if the supervisor in this office got a lot of training on
management.
34. Having a great team to work with makes my job enjoyable and working toward a
common goal helps motivate me to do my job well.
35. The Division is managed and run much better now than it was under the previous
leadership, where personal agenda's were priority and key positions were filled
with staff who were not suited for the positions, which had a huge negative
impact on how the Division did business. Accountability measures have been
put in place, employees who do not meet the minimum standards are dealt with
accordingly. Some sections have been restructured to promote more efficient
and effective services are being provided.
36. I have worked in government services for many, many years and without a doubt
this is the finest organization with which I have ever been associated. The old
adage, good enough for government work is one that turns my stomach and is a
standard in other states and probably other areas of this state, but it is certainly
not a mantra in this Division. This Division is a shining star and is doing an
incredible job for the visually impaired of Florida.
37. My immediate supervisor is strongly supportive of her staff, but does not get the
same support from her supervisor. When doing surveys like this terms such as
management should be defined. Who is management versus who is a
supervisor.
38. It is a high time of change and staff fear for their jobs.
39. The morale has steadily declined over the last two years. There have been
issues with ethics violations regarding how the students and staff have been
treated.
40. Staff and upper management do not feel free to speak up when they have a
differing opinion. This results in low morale and discourages thoughtful
discussions on how to develop contracts with providers and provide oversight to
ensure our clients are being served in the best way possible. This also results in
an environment that is fraught with fear which discourages staff from taking
ownership because a mistake can be detrimental to your job. The amount of
turnover at the upper management level has been quite high and is often done
as a knee jerk reaction after an incident. The type of quick and unexpected
change causes too much instability in an already fragile environment. The
actions appear emotionally and personally based rather that a result of a
thoughtful decision. Opinions on individuals are based on personal feelings
rather than that individual's value to the Division. The Division could flourish from
strong leadership and someone that truly leads rather than someone who thinks
she is leading because no one speaks up. We need a leader who will make
decisions, not someone who often says I don't care how we do it and once a path
is taken, then decides it should be done another way.
41. Manager spends too much time just chatting with people in their offices.
42. Sharing of best practices statewide, and success stories of clients and their
employers.
43. The Director is not a good leader she set people up to fail.
44. I think that IG should look at hiring practicing and management styles from the
state office which filters down to each of the offices in the field.
45. We all need our jobs. I hope that this survey sheds some light on the problems at
hand.
46. Provide raises and promotions based on honest to goodness merit. Do not create
positions and pay acquaintances and friends high salaries, and then say that
dollars aren't there to pay actual working staff more money. A boss, a Director
should make agency wide decisions, and allow the experts who are in position to
do what they are there for. Learn your work environment and where staff work
and just say good morning to them instead of only to management and those
who are in offices. If you have to constantly go to a staff member to learn how to
do your job, acknowledge them and if possible pay them for their actual worth.
47. The improvement made in this division the last three years is phenomenal!
48. I believe the Governor Scott is the worst we have had and he needs to resign. A
good look needs to taken into the use of personnel in DBS and questions asked
about why we don't provide technical support for clients in the work place. Why
is there not a position for a Rehab Engineer? Why are we allowing Private
Facilities to provide so much service to clients when we have a facility that the
state has paid for?
49. Policies from Tallahassee discourage any job performance that would remotely
assume that the people whom they have hired are actually competent
professionals who are interested in excellence. The whole approach seems
been designed to corral a few bad apples while insisting that everyone else
SHOULD be thrilled with just doing the bare minimum in their jobs to get by.
Why not just fire the people who aren't doing what they are supposed to be doing
and re-instating the idea that counselors are professionals and capable of writing
a program plan?
50. When asked above about being proud to work for DBS, if you are asking am I
proud of working with the blind community, my answer is a very proud YES. Am
I proud to be a part of the positive encouragement given to the blind community
that the only part of themselves that has changed is their vision but they still
possess the abilities to be successful, they just have to learn through the Rehab
Center, new methods to transfer those abilities to their current physical
challenge. I believe all blind persons can be employed if they truly want to be
employed. There are jobs to be created. There are employment needs in
Florida yet to be discovered and filled and I believe if equipped, the Rehab
Center can train the blind population to fill those jobs. What I am not proud of in
working with DBS are the restrictions Tallahassee Management and our current
Bureau Chief puts on the Rehab Center through outdated equipment teaching
materials, and encouragement to our exceptional staff who have ideas of how to
bring the blind community into the future needs of Florida but current
management lacks any comprehension or desire to hear these ideas so our staff
have smothered their creativity in response to the current management's
attitudes.
Management Practices
Responses to Question 42. Please provide suggestions in the input box below that you
think would improve management practices.
1. Recognize that the agency's biggest asset is employees. Establish an inclusive
workplace that energizes the people who fuel our agency's success. It is
essential to recognize that the relationship between people and leaders is based
upon mutual benefit. It is not paternalistic, but instead is based on individual
engagement and contribution.
2. Be Fair.
3. Make sure that a position is offered to all qualified personnel within the facility.
Perhaps be sure that the position is also advertised outside of the facility to
hopefully prevent hiring based on the one deciding on hiring having known the
person and their family. Also avoiding hiring only because the person is younger
and should therefore be around longer than other more qualified staff.
4. Sometimes it seems the hiring process takes long, but that may be because the
qualifications are not met.
5. Change for change's sake causes low morale and anxiety. We are put on an
important project for a week or 2, then priorities change and it is dropped, and
another is dropped in our laps. All the while regular routine duties are neglected.
6. Accept people for who they are; tolerate their individualism. Supervisors could
not perform to the best of their abilities if DAs have the monopoly of every
decision and allow Supervisors to only sign where indicated.
7. There has been excessive turnover in DBS over the last couple of years. Many
staff fear for their jobs daily. I have been told on several occasions by my
superior that I need to keep off the radar, that any of us could be gone in a
second depending on the mood of management. Over the last few years more
and more money has gone into contracts with CRP's and more and more of the
job duties DBS staff did are going to the CRP. We can no longer provide the
training devices, low vision evaluations and more to our client's because the
money we once had is going to the CRP's.
DBS cannot do anything, (create a form, develop a business process, etc)
without involving the CRP's in what we are doing. It's as if we work for the CRP
not for DBS. The unwritten understanding in the agency is that the only way to
keep your job is to make sure your CRP likes you as management has given
every impression that their support is with the CRP, not DBS staff. Management
needs to be much more supportive of DBS staff and help DBS grow and develop,
not continue to dismantle the agency, or at least that is how it feels.
8. It feels that contract providers are running DBS at times. It also seems like we
don't value our employees and/or hire people that are familiar with the business
of rehab.
9. Positions are filled without our knowledge and just between Management and the
person being hired. We see new people and don't even know their
names...friends are being hired from outside...a lot of favoritism going on that is
not very fair.
10. I think that mid management has its hands tied by statewide policies, such as
hiring freezes that won't permit replacement of employees who leave or not
permitting to pay a reasonable wage, so to attract appropriate workers. How can
we expect to hire qualified people, when anyone pays better than us?
11. Management practices shouldn't promote an office environment, nor reflect an
image that if you do not belong to their group, very clickish, you are not going
anywhere?
12. Sensitivity training.
13. DA does not know everything, especially when he only been on the job less than
2 years.
14. Do not be dismissive of employees input or concerns.
15. With such low salaries for state employees, especially for those of us in human
services, our job performance and professionalism are better than could be, or
should be, expected. The clients we serve, and those of us who serve them, are
not valued enough by those who govern us.
16. Hire competent management; supervisors should be firm, but yet flexible, fair and
understanding. Must be good communicators.
17. New director
18. The State needs to consider raises for the employees.
19. It may be time for senior management to move on and be replaced with
individuals who can restore confidence in the agency mission. Senior
management can focus more on administrative, planning, motivational and
political issues of the organization. Let trained staff be responsible for the
programmatic functions of the agency.
20. This section is difficult to respond to because I need to consider the specific
entity with which my office contracts, which does a good job, as well as another
entity in my district, which I know does a poor job. The contract monitoring for
deliverables does not take into consideration how many clients may not have
received services even though the deliverables may technically have been met.
Additionally, there has been a huge shift in money from DBS to contracting
entities, as well as a shift in responsibility from DBS to those entities. This has
caused a significant change in work flow and responsibilities for some DBS
employees. And, in some cases, contracting entities do not appear to have
geared up for those new responsibilities, but rather have seen them as simply
new cash flow streams, without providing sufficient staffing to do the work
involved.
21. Director Joyce Hildreth has always been extremely courteous and helpful to me.
I find her to be very knowledgeable and compassionate about her job and the
DBS mission. She has taken the time to ensure I have the knowledge to succeed
in the organization. Our Deputy Director is knowledgeable about contracts and
different programs.
Unfortunately, she is very critical and on a number of occasions has openly
criticized me and other employees in meetings and on conference calls. She has
made negative comments re. the Director and other personnel that have made
me feel uncomfortable. She is not receptive to change or ideas from others. I
would encourage her to be a better listener and not make faces, rolling of eyes,
and waving hands when she is not in agreement. I would like to see her trust the
employees she has working for her. My immediate supervisor is knowledgeable
on the contracts side. She is a very positive, supportive, appreciative person. I
would only recommend she be more receptive to change and not be stuck on
business as usual. Finally, the Bureau Chief of Client Services manager should
share her management style with the team. She is an active listener and a
pleasure to work with.
22. I have not experienced anything in my current office. But for several years I was
forced to work in an extremely hostile work environment. Even though it was
reported little to no action was taken to stop it. Also in the process of being
promoted. My former DA was allowed to say lies and slanderous things against
me with no repercussions.
23. Whenever individuals in the agency speak up about poor performance from
contractors, they are severely reprimanded and often they are threatened that
they will lose their job. This has personally happened to me on more than one
occasion.
24. Good God man we only fire or take resignations here. Positions are frozen and
never coming back. We will be down to half of the employees here by the end of
the year.
25. Hire qualified people. There are a lot of hard working people in the ranks that no
one ever hears about who do a lot of work for little pay, no glory for them either.
26. Very well organized with effective results.
27. The only time I have ever felt I was subjected to an inappropriate conversation
was with someone outside of the Division of Blind Services who gave the
appearance of pulling rank and micro-managing where he really had no authority.
28. If staff are asked to complete a task, which is then approved by the supervisor,
management needs to discuss changes instead of making them without
consulting the people who did the work. We are made to feel that our work has
little or no merit. Management needs to communicate clearly with staff and not
make changes arbitrarily. Changes need to be well thought out, discussed with
concerned employees before being put into practice.
29. People do not know what is going on or to expect. Probably no one higher does
either so why burden staff with supposition.
30. Hire competent management. Take courses in leadership management. Take
lessons in communication. Respect staff. Be consistent
31. An environment where staff can openly express differing opinions and not fear
loss of job.
32. It has taken up to two months to get approval to go ahead and input purchase
orders for goods and services that we need. A more timely response would help
greatly in meeting our needs.
33. Better leadership.
34. The CRP does not always provided services in a timely manner or cost effective.
35. Hiring should be done on ability, skills and not on favoritism. Such as hiring from
A CRP just because.
36. Let employees do their jobs and do not be so critical. We are all experiencing
tough times and should be lifting each other up and not throwing some people
down.
37. Director has freely admitted in front of staff that she holds grudges and acts on
them. Personnel management speaks about firing staff with her door open.
Personnel Management tells staff that she has fired eighteen people this year
and doesn't mind terminating more. Statement made early December of 2011. I
well understand the role of the state office or central office, but based on actions
and words, the goals and objectives of current upper management confuses and
distorts this role as well as the spirit of positive and effective leadership.
For the most part policies and procedures are up to date and sensible given the
agency's mandate; however, some policies are ambiguous at best and harmful to
the agency and client at worst. Because such a large amount of funds are
allocated to outside contracts by way of CRPs, there are fewer funds to follow
through with other needed services. This is also hurtful because these contracts
cover a bulk number instead of individuals. Contracts are loosely monitored but
not the actual CRPs. and if staff complains about CRPs or otherwise find them in
wrong doing, CRP director calls DB Director and corrective actions are brought
against staff. This could be and has been anything from a taking away of duties,
to demotion, to termination. Investigate demotions and terminations of pass two
years.
Some contractors are outstanding, but others are not. This is why the blanket
contract to CRPs in every district without much recourse is a serious problem.
When its 6 months after a person is hired and other staff is still doing their job
that they were supposedly hired for, this is a concern, most especially when new
staff was hired at an exceedingly high pay grade. Check the last ten new hires in
the state office, in particular under the directors office.
Memos and bulletins go out, but upper management knows who the new hire is
before the closing date for submission of applications. Also office staff who
would be a great filler of said position is completely ignored while the new hire
makes anywhere from $10,000 to $15,000 more than staff who fills that same or
similar position. Not true unless upper management sees you as one of theirs.
Sorry, but this is how things are in the state office.
It is clear that minorities do not generally receive raises or get hired in high
paying positions or management, rather or not he or she manages staff
unofficially. That I can remember, The only time I was subjected to conversation
that made me feel uncomfortable was when a member of management, a bureau
Chief stated that he didnt like particular client and so wanted him out of the BEP
Program.
Upper management practices do not promote, perpetuate, or maintain good
organizational success because it is extremely divisive. Staff are asked not to
communicate with colleagues of other sections unless work directly demands it,
and when staff who everyone knows does a great job at what they are hired for
receives an average evaluation and others most especially management
receives high marks, morale is understandably low.
38. I am not aware of any inappropriate hiring but one should really look at the
turnover of positions and constant moving around of positions and ask what is
going on. As far as chances for promotion within I don't think that is promoted at
all. I definitely feel I was cheated out of a promotion and the private facilities
gained while services to clients in the employment sector have suffered greatly.
39. Yes, positions are filled with qualified people who are then immediately
discouraged from seeking to do the excellent work of which they are capable by
taking away their independent status and having someone looking over their
shoulders every second. If DBS is so concerned about improving their RSA
reviews, why not tie independent status into things like how many times
deadlines for eligibility decisions or plans have been missed? Wouldn't how
many times a plan has been returned to the counselor because of errors be a
better way of approaching the issue of the supervisor approval issue than a
blanket no one can make decisions about a plan, no matter how good a
particular counselors' plans actually are?
The way it is now, good counselors don't even have a means by which they
could be rewarded for good work by re-gaining independent status. With regards
to the question related to cost effectiveness earlier in this survey it is not only
extremely time consuming for the counselor to have to wait for supervisory
approval on every plan, it is also hard on the clients to have to meet again with
the counselor to sign off on plans that could have been done in one session. I
often wonder how it must look to our clients to have to get supervisory approval
for decisions that he/she and his/her counselor have made.
40. When giving performance appraisals supervisors are limited on how high a grade
they give staff. This is wrong. We just had our performance appraisals and we
were told Supervisors were told by our DBS Director to limit giving staff a 4 on
their job responsibility as this could involve money given the staff person. Some
of us were given a total that was under a 4 for that reason even though we
exceed our standards. On paper, you won't see the reason we were only given
3's is because the supervisor was hindered in giving staff 4's but it makes us
appear as average. This is not always the case. I expect my performance
appraisal to reflect my efforts. I expect the best of myself as some others do of
themselves at the Center and it is a slap in the face to be required to accept less
because it could involve money. Staff moral plummeted and some refused to
sign their performance appraisals.
41. Honesty about goals for the office; a more straightforward manner; less secrecy
42. Replace Joyce Hildreth as the director of DBS!! We need a change and an
individual that truly cares about blind Floridians and their well being!!
43. I have been in some conversations with management that make me
uncomfortable. They typically involve reviewing data for reports DBS supplies to
Legislature, State, and Federal government. Not often, but occasionally, our
management will gloss over some inaccuracies in reporting because it's the most
expedient or easiest thing to do rather than do the hard work it takes to ensure
integrity in reporting. Management seems concerned that correcting an issue in
reporting in the current year will be significantly different than the previous year's
submission and draw attention to the agency. I prefer to employ sound and
honest management practices, own up to mistakes and be able to explain the
differences if necessary. As for contracting, I believe we need to be more diligent
in monitoring and managing our contractors. We could be more prudent and
responsible in how we measure our contractors' performance and how we
reward them.
44. My direct supervisor and Bureau Chief both exhibit a serious lack of integrity, and
extremely poor management practices. They are often hostile, and publically
demean and belittle employees. Discrimination is rampant, and the handicapped
and visually impaired are flagrantly discriminated against in two ways. (1)
Equipment needed for communication by the visually impaired (and provided by
a previous administration) has been confiscated as unnecessary toys. (2) The
incompetence and inefficiency of the head of production is supported by the fact
that audio recordings are more than a year behind, sporadic at best, yet he is
praised and rewarded by the Bureau Chief as a brilliant shining star. What this
means is that patrons of the library do not receive audio recordings of serial
magazines, which is also discriminatory.
45. Incentives for job placement for our clients are provided to CRP's instead of DBS
employees. A $100.00 bonus would be a great incentive for Counselor's to place
their client's in job....instead DBS will pay a CRP $4000.00 per job placement.
46. We need a manager who knows how to manage
47. Change in management from the top down. Anything has to be better than what
we have now.
48. Management needs to be open, honest and reflect the mission of DBS to be a
strong leader. Imposing fear on staff does not a good Manager Make. 2. The
Entire Management Team should be accessible, and staff should not fear
retaliation for any problem, opinion, or issues brought to their attention. 3. Staff
needs to be told the true Goals and Objectives of their agency. 4. Management
must model the behavior expected of staff. 5. Policies and procedures are clear
as written, but Management needs to use them as a guide, as they are written
and not a weapon for intimidation. 6. All vacancies should be made public,
advertised through People First, and all current staff given the opportunity to
apply. 7. Bureau Chiefs that handpick individuals outside of the agency give the
appearance of misconduct. Who are these people? How did they come to be
employed in a State Agency without the position being advertised? Why is this
permitted to happen over and over again. The Director is allowing this to happen
and is giving staff the appearance of approval for this type of action. This is not in
the States policies and procedures. If staff must adhere to these, shouldn't
Management also? 8. Management practices do not promote organizational
success, they shed no light on the true mission and success that could be.
49. Personally I feel that anyone who is a DA must have knowledge of the clients
who are served by the agency, must have a direct idea of what the agency
stands for as well as an active job history of vocational rehabilitation.
50. If the HR person does not like you - you will not have an opportunity for
advancement nor for raises.
Florida Department of Education
Division of Vocational
Rehabilitation
Climate Survey: Final Report
February 18, 2011
2010
2010 Employee Climate Survey Results
Final Report
TABLE OF CONTENTS
List of Abbreviations .................................................................................................................................................................................. 4
Executive Summary ................................................................................................................................................................................... 5
.
Comparison of Current Climate Items with Previous Climate Items ......................................................................................................... 6
Quadrant Analysis .................................................................................................................................................................................. 8
Factor Analysis ........................................................................................................................................................................................... 9
General Findings and Recommendations ................................................................................................................................................ 10
Interpretation of Statistical Comparisons ............................................................................................................................................ 11
Demographics .......................................................................................................................................................................................... 12
"Average Satisfaction" Score and "Percent Agree" Across Select VR SubUnits ..................................................................................... 18
Overall Satisfaction: Level of Agreement ................................................................................................................................................. 21
Interpretation of Charts ....................................................................................................................................................................... 21
Entire VR .............................................................................................................................................................................................. 22
DVR Headquarters ............................................................................................................................................................................... 24
Bureau of Administrative Services ....................................................................................................................................................... 26
Bureau of Field Services ....................................................................................................................................................................... 28
Bureau of Partnerships & Communications ........................................................................................................................................ 30
.
Office of the Director ........................................................................................................................................................................... 32
2
Bureau of Rehabilitation and Reemployment Services (BRRS) All ......................................................................................................... 34
BRRS District 2.................................................................................................................................................................................... 36
BRRS District 4.................................................................................................................................................................................... 38
BRRS District 6.................................................................................................................................................................................... 40
Bureau of Field Services ........................................................................................................................................................................... 42
Field Services (HQ and Area Offices) .................................................................................................................................................... 42
Field Services (All Areas) ...................................................................................................................................................................... 44
Area 1 ................................................................................................................................................................................................... 46
Units 01A/ 01B/ 01C/ 01D/ 02A ........................................................................................................................................................... 48
Units 03A/ 04A/ 5A/ 05B/ 05C ............................................................................................................................................................. 49
Area 2 ................................................................................................................................................................................................... 50
Units 07A/ 07S/ 08A/ 08B/ 08C/ 08D/ 08F/ 08G ................................................................................................................................. 52
Units 08E/ 09A/ 09S/ 10A/ 10B/ 11A/ 11B .......................................................................................................................................... 53
Area 3 ................................................................................................................................................................................................... 54
Units 12A/ 12B/ 12C/ 12E/ 12G/ 13A/ 13B .......................................................................................................................................... 56
Units 12D/ 12F/ 17A/ 19A/ 20A ........................................................................................................................................................... 57
Area 4 ................................................................................................................................................................................................... 58
Units 14A/ 14B/ 14C/ 14D/ 18A/ 18B .................................................................................................................................................. 60
Units 15A/ 15B/ 15C/ 15D/ 15E/ 16A/ 16B .......................................................................................................................................... 61
Area 5 ................................................................................................................................................................................................... 62
Units 21A/ 21B/ 21C/ 24B/ 24D ........................................................................................................................................................... 64
Units 22A/ 22B/ 22C/ 22D/ 24A/ 24C .................................................................................................................................................. 65
Area 6 ................................................................................................................................................................................................... 66
Units 23A/ 23B/ 23C/ 23E/ 23F/ 23H/ 23M ......................................................................................................................................... 68
Units 23D/ 23G/ 23J/ 23K/ 23L/ 23N ................................................................................................................................................... 69
Content Analysis ...................................................................................................................................................................................... 70
Notes Page ............................................................................................................................................................................................... 78
3
LIST OF ABBREVIATIONS
Bureau of Administrative Services
Bureau of Compliance and Oversight
Bureau of Field Services
Bureau of Partnerships and Communications
Bureau of Rehabilitation and Reemployment Services
Division of Vocational Rehabilitation
Headquarters
Office of the Director
BAS
BCO
BFS
BPC
BRRS
DVR
HQ
OD
4
EXECUTIVE SUMMARY
During November and December of 2010, the Florida Division of Vocational Rehabilitation (DVR) conducted a division
wide employee survey of factors that affect employee satisfaction and work climate. DVR used online survey software
designed to ensure confidentiality of employee responses. The VR Employee newsletter and emails were used to notify
DVR employees of survey information and distribute a secure link to the survey. The survey was available to all DVR
employees, and a response rate of 67.4% was achieved.
The survey consisted of 35 climate items, which respondents rated on scales of agreement and importance.
Respondents had the opportunity to provide written feedback regarding climate items. The survey also included 2
openended questions requesting feedback on DVR climate factors that most satisfied, and most dissatisfied,
respondents.
A total of 86.12% of DVR employees indicated that they strongly agree or somewhat agree with the statement "Overall, I
am satisfied with DVR as a place to work." Preliminary analysis indicated the highest percentages of agreement
(strongly agree plus somewhat agree) include the following climate items. Table 1 in this report includes the results of
all items.
I am held accountable for the quality of my work. (97.84%)
I can identify what I must do to be successful at my job. (96.81%)
I know what it takes for me to meet the expectations of my customers. (96.81%)
I have the education and training necessary to effectively do my job. (96.28%)
Additional analysis indicated that the following items have the greatest potential to improve overall employee
satisfaction. Additional information is available on Table 2 of this report.
Managers, supervisors, and other leaders encourage innovative or new approaches to do things at work.
Work in my immediate unit is organized to promote cooperation.
My opinions seem to count when I provide them at work.
My supervisor promotes a work environment that fosters trustworthy relationships.
The climate survey team is conducting further analyses that will provide better insight into DVR employee satisfaction
and assist managers to make informed decisions on actions to take for further improvements. If you have any questions
regarding the climate survey, please feel free to contact a member of the Continuous Improvement Unit.
Steve Collins
Libby Moody
Joshua Durden
Russell Hellein
Carmen Dupoint
5
850.245.3429
850.245.3281
850.245.3331
850.245.3300
850.245.3299
steven.collins at vr.fldoe.org
libby.moody at vr.fldoe.org
joshua.durden at vr.fldoe.org
russell.hellein at vr.fldoe.org
carmen.dupoint at vr.fldoe.org
COMPARISON OF CURRENT CLIMATE ITEMS WITH PREVIOUS CLIMATE ITEMS
Table 1 below shows the results for all survey items in comparison with climate surveys conducted since 2003. The far
right column indicates percentage point changes from 2007 to the current survey. It should be noted that 22 of 27
survey items (81.5%) show improvement from the 2007 survey. Also of note is that 4 of the 5 items that did not show
improvement had negligible drops in performance from the 2007 survey. More detailed analysis of all survey items and
factors is currently underway.
Table 1: Comparison of Current Climate Items with Previous Climate Items
Item
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Statement
DVR Headquarters leadership has set high performance expectations.
DVR's strategic plan goals and initiatives have been communicated to me.
I understand how my individual performance contributes to DVR's success.
DVR's organizational values are promoted by Headquarters Leadership.
Overall, decisions by DVR Headquarters Leadership are based on that which
most likely advances our mission and goals.
My supervisor provides clear and consistent expectations regarding my work.
I am able to get the information, questions answered, decisions, etc. necessary
to effectively do my job.
DVR communications, such as the DVR Employee Newsletter and VR-Inet,
enables me to stay informed of strategies and events within DVR.
My supervisor promotes a work environment that fosters productivity.
My supervisor promotes a work environment that fosters trustworthy
relationships.
Managers, supervisors, and other leaders encourage innovative or new
approaches to do things at work.
I am able to use my skills and initiative to perform my work.
I know what it takes for me to meet the expectations of my customers.
In my immediate work unit, we regularly use data and information to improve
our work unit's performance.
I have the equipment and resources necessary to perform quality work.
I can identify what I must do to be successful at my job.
I am held accountable for the quality of my work.
I am comfortable being held accountable for the quality of my work.
In my immediate work unit, employees who perform well have opportunities for
recognition for their accomplishments.
In my immediate work unit, employees who perform well have opportunities for
increased pay.
New employees are given a helpful orientation to DVR.
I have the education and training necessary to effectively do my job.
Someone at work encourages my development.
There is a clear system to take advantage of opportunities for career
advancement.
Work in my immediate unit is organized to promote cooperation.
My workload enables me to balance my work and personal life.
Employees are involved in decisions which may affect how their work is
performed.
My opinions seem to count when I provide them at work.
VR provides employees who perform well opportunities for advancement.
I feel that I am part of a team at work.
My supervisor, or someone at work, seems to care about my success.
I feel physically safe and secure at work.
Ethical behavior is promoted by Headquarters Leadership.
The feedback I received from my most recent performance review was useful
(directed towards helping me improve and/or advance).
Overall, I am satisfied with DVR as a place to work.
2003
2005
2007
2010
Change
between
2007/2010
6.70%
3.56%
5.04%
2.11%
7.62%
0.54%
1.13%
6.13%
6.96%
2.32%
2.11%
0.89%
0.21%
NA
4.18%
2.78%
NA
2.90%
7.20%
11.71%
NA
3.03%
3.15%
NA
NA
NA
NA
1.11%
15.71%
3.86%
1.51%
NA
6.23%
0.67%
8.73%
76.30% 84.46% 91.16%
66.80% 80.20% 84.46% 88.02%
73.90% 82.00% 90.11% 95.15%
71.70% 82.57% 80.46%
62.70% 76.30% 83.92%
74.80% 79.90% 84.46% 83.92%
75.00% 81.32% 82.45%
59.30% 77.20% 83.99% 90.12%
69.70% 75.67% 82.63%
69.70% 75.67% 77.99%
49.60% 55.50% 67.50% 69.61%
90.11% 89.22%
89.20% 96.10% 97.02% 96.81%
83.46%
62.50% 61.40% 74.88% 79.06%
88.60% 91.50% 94.03% 96.81%
97.84%
84.40% 96.50% 93.09% 95.99%
18.00% 58.08% 65.28%
18.00% 25.75% 14.04%
69.50%
73.10% 92.10% 93.25% 96.28%
68.40% 67.80% 73.94% 77.09%
60.06%
78.16%
77.69%
64.74%
68.30% 68.00% 70.96% 72.07%
18.00% 36.73% 52.44%
75.20% 78.34% 82.20%
66.40% 72.20% 80.38% 81.89%
83.78%
71.70% 82.57% 88.80%
62.30% 68.80% 75.67% 76.34%
66.30% 66.50% 77.39% 86.12%
6
Comparison of Florida Division of Vocational Rehabilitation Employee
Satisfaction to Other Public Agencies
Overall
Satisfaction
Rate
Survey
Organization
Year
State of Vermont
95%
2008
Florida Department of Transportation
93%
2010
Florida Division of Vocational Rehabilitation
86%
2010
Georgia Perimeter College
78%
2009
Federal Government (overall)
72%
2010
State of Georgia
71%
2009
Florida Department of Health
68%
2008
Items with Greatest Potential for Improving Overall Satisfaction
Item responses were analyzed using a quadrant analysis approach. This approach is a standard best practice for
analyzing climate data. Table 2 below ranks the items that have the greatest potential to improve overall employee
satisfaction. For example, Item 11 ("Managers, supervisors, and other leaders encourage innovative or new approaches
to do things at work.") suggests that the DVR leadership team should continue to deploy innovative and new approaches
for doing work. This may be addressed by expansion of performance improvement projects such as Rapid Process
Improvement events. It should be noted that this is a preliminary finding and that additional analyses will be necessary
prior to taking action on the results.
Comparison of Current Climate Items with Previous Climate Items
Items with the Greatest Opportunity for Impact
Item
Statement
Managers, supervisors, and other leaders encourage innovative or new
approaches to do things at work.
Work in my immediate unit is organized to promote cooperation.
My opinions seem to count when I provide them at work.
My supervisor promotes a work environment that fosters trustworthy
relationships.
In my immediate work unit, employees who perform well have opportunities for
recognition for their accomplishments.
Employees are involved in decisions which may affect how their work is
performed.
There is a clear system to take advantage of opportunities for career
advancement.
Someone at work encourages my development.
VR provides employees who perform well opportunities for advancement.
New employees are given a helpful orientation to DVR.
2003
2005
2007
2010
Change
between
2007/2010
2.11%
NA
1.11%
2.32%
7.20%
NA
NA
3.15%
15.71%
NA
11
25
28
10
19
27
24
23
29
21
49.60% 55.50% 67.50% 69.61%
78.16%
68.30% 68.00% 70.96% 72.07%
69.70% 75.67% 77.99%
18.00% 58.08% 65.28%
64.74%
60.06%
68.40% 67.80% 73.94% 77.09%
18.00% 36.73% 52.44%
69.50%
7
QUADRANT ANALYSIS
Quadrant Chart by Item Number Items that fall in the Northwest Quadrant are most important to overall
satisfaction, but have the lowest scores.
0.60
Northwest Quadrant lower score/ higher
importance
11
28
25
10
Higher score/ higher
importance
30
7
9
31
23
34
6
21
4
5
12
0.50
27
24
29
19
24
29
Relative Importance to Overall Satisfaction
33
0.40
26
15
14
2
8
32
3
Lower score/ lower importance
0.30
16
20
17
18
20
1
13
22
0.20
0.10
Higher score/ lower
importance
0.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
2.25
2.50
2.75
3.00
3.25
3.50
3.75
4.00
Average Satisfaction Score
The further to the left ("West") boxes are, the less satisfied respondents were with that item. The higher boxes go in the
chart ("North"), the more important that item is to overall satisfaction. Boxes in the left upper corner ("Northwest")
have the most potential to improve overall satisfaction, because they have relatively greater impact on it and people are
relatively less satisfied with that item (making improving it easier).
8
FACTOR ANALYSIS
A factor analysis was run to better understand what influences satisfaction at VR. Factor Analysis generates factors
sets of related items that explain an individual's response to the survey questions. Variables were listed on the factors in
their relative order of importance. Two factors were particularly important. By far the most powerful factor was
associated exclusively with Supervisors' impact on their immediate work environment. It was the key factor explaining
overall satisfaction and most of the elements of the quadrant analysis. This suggests that efforts to improve satisfaction
in VR (either directly or indirectly through changes in the quadrant analysis items) must have a significant impact on the
immediate work environment to be successful. A second factor was heavily associated with leadership driving a positive
view of the agency. This suggests that efforts to improve satisfaction should be geared in part to improving the image of
the agency in employees' eyes.
Factor
Supervisor Impact
on Immediate Work
Environment
#
*10
9
6
31
30
*28
*25
*23
7
*11
34
*19
*27
35
12
14
Leader's Impact on
Employee's View of
VR
1
3
4
33
5
2
Survey Item
My supervisor promotes a work environment that fosters trustworthy relationships.
My supervisor promotes a work environment that fosters productivity.
My supervisor provides clear and consistent expectations regarding my work.
My supervisor, or someone at work, seems to care about my success.
I feel that I am part of a team at work.
My opinions seem to count when I provide them at work.
Work in my immediate unit is organized to promote cooperation
Someone at work encourages my development.
I am able to get the information, questions answered, decisions, etc. necessary to
effectively do my job.
Managers, supervisors, and other leaders encourage innovative or new approaches.
The feedback I received from my most recent performance review was useful (directed
towards helping me improve and/or advance).
In my immediate work unit, employees who perform well have opportunities for
recognition for their accomplishments.
Employees are involved in decisions which may affect how their work is performed.
Overall, I am satisfied with DVR as a place to work.
I am able to use my skills and initiative to perform my work.
In my immediate work unit, we regularly use data and information to improve our work
unit's performance.
DVR Headquarters leadership has set high performance expectations.
I understand how my individual performance contributes to DVR's success.
DVR's organizational values are promoted by Headquarters Leadership.
Ethical behavior is promoted by Headquarters Leadership.
Overall, decisions by DVR Headquarters Leadership are based on that which most likely
advances our mission and goals.
DVR's strategic plan goals and initiatives have been communicated to me.
*Items denoted in bold have the greatest opportunity for impact
9
GENERAL FINDINGS AND RECOMMENDATIONS
Finding
Significant differences in education, age, and length
of employment.
Recommendation
Focus on training, career development, and
succession planning.
HQ significantly more satisfied than Field.
Focus on open communication and behavioral
demonstration of values.
Supervisors have a major impact on work unit
climate.
Focus on behaviors that build trust. Review content
analysis for ideas. Engage employees in identifying
solutions. Allow for autonomy, clear expectations,
and adequate information to do the job.
Comments indicate some offices lack adequate
basic work equipment, poor layouts, inadequate
privacy, etc.
Review physical office environment and equipment
needs; then align plans and budgets to meet needs.
10
Audit No.: M-11/12-20
Prepared By: ___KK _____
Date: ___5/3/12______
DBS Management Review Interviews
Reviewed By: __________
Date: __________
We have included interviewee names; however, we ask that you
keep names private due to the sensitive nature of this review and concerns of
retaliation.
Twelve interview write-ups were judgmentally selected for your review. The sample
consists of two former employees and ten current employees five state office and five
district office.
We feel the sampled interviews are representative of our results and show the polarity of
interview responses. The interviews address topics such as:
· Working environment,
· Turnover concerns,
· Contracts and monitoring,
· Relationships with the Community Rehabilitation Providers (CRP),
· Hiring process, and
· Communication.
Approximately 35 additional write-ups are available, should you wish to see them.
Management Review of the Division of Blind Services
Audit No.: M-11/12-20
Page 1 of 1
DBS Management Review
OIG 11/12-20 M
Person(s) Interviewed/Title:
Date: 3/6/12
1)
DBS position/ role?
Interviewer: JM
Notes prepared by: KK
Senior Management Analyst II with the Bureau of Client Services.
Job duties: submit federal reports (quarterly and annually); budget allocations to
Districts for program activities; and quality assurance activities.
2)
How long have you worked at DBS?
Since September 2010, 1.5 years. Came from the Division of Vocational
Rehabilitation (DVR). Was an internal auditor with DVR.
3)
How long have you worked in your current position? Who is your supervisor?
1.5 years. Supervisor is
4)
(Bureau Chief for Client Services).
How would you describe the working environment within DBS/ your office?
Within his Bureau: Positive. Running as well as it can. Bureau Chief does a very
good job, trying to keep up the best they can based on inherited activities.
Within Division: Lot of micro-managing. Division Director makes all decisions. Staff
decisions can be overturned. Could be fear in making decisions (retaliation),
especially when dealing with the CRP's. Division Director has a tight relationship
with the CRP's too tight. (He has heard this from others). Thinks one could make
the argument that there is preferential treatment.
High turnover at high levels of management.
Questioned if that was
normal/reasonable. Thinks turnover is due to staff opposing the Division Director.
Let go if they are seen as an "impediment" to what the Division Director wants
done. He questioned who enabled turnover. Said Linda Champion and Sheila
White sign off on personnel actions, so should they also be held responsible. He is
not sure about Sheila White's relationship in the Division.
No specific names regarding retaliation.
5)
Can you name any positive role models or areas that have improved within the
Division?
DBS Management Review
OIG 11/12-20 M
-
Bureau of Client Services
Timely submittal of federal reports
Allocations to Districts / Budget allocation process
Better job oversight in District offices
Better job handling client appeals (not handled in Districts and moved up to
State office)
6)
What is your impression of the staff turnover that has occurred over recent
years?
Too high given the size of the Division. Higher than at DVR. Some people forced
out/to resign.
7)
Do you feel DBS contracts are properly negotiated and managed without
preferential treatment?
Doesn't have direct involvement in this area. Thinks contracts are a sore point with
the Division and Division Director.
Possibility of State not getting what it is paying for. Lack of monitoring and
oversight. Don't look too deep at CRP's. Don't want to apply sanctions. Gave an
example of concerns with the number of clients served. Believes CRP's are paid
the same if they serve 1 client vs 100. Staff in contract monitoring unit doesn't feel
comfortable about staff turnover. Review docs/CRP invoices, but don't question.
Just process it.
8)
Do you feel the needs of DBS clients are being met effectively?
Could compare and contrast the money given to Districts vs. CRP's. Hard to
document qualitative aspects of reallocated money to CRP's from Districts. Would
want to look at quality and number served.
Jim said someone else mentioned that more funds to contracts could mean less
funds for direct services by the districts. David agreed with this statement.
9)
Are DBS contract managers and staff allowed/ authorized to effectively manage
contracts with CRPs?
Not in accordance with DFS rules.
10)
Do you feel that DBS funding is being put to its best use?
Issues with contracted services. Not sure the Division is getting the best bang for
DBS Management Review
OIG 11/12-20 M
its buck. Not verifying numbers (served). Only desk reviews, not onsite visits.
Afraid to ruffle the feathers of CRP's because it will get back to the Division
Director.
11)
Is the DBS hiring process fair?
Not exactly. People hired that don't have minimum credentials (college degree)
because they are a friend of someone. Internal staff does not think they have a fair
chance for advancement.
12)
Do you have any sense of the presence of favoritism in DBS hiring practices?
Not getting best/brightest people for positions. Questioned sign-offs by Linda
Champion saying things don't happen in a vacuum.
When asked how to find favoritism, David suggested we look at salary
enhancements (more than one) and turnover over the last two years.
13)
Do you feel DBS employees are free to express themselves without concern?
No.
14)
How would you describe communication from and with executive management?
Operating in silos. Some staff are favorites of the Division Director and feed her
information.
Would like more teamwork.
Not most positive environment compared to other work experience with DOE.
Fear of retaliation if list of interviewees gets back to Division Director. No doubt of
retaliation of interviewees by executive management if the list gets out. Requested
we not disclose names.
15)
Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
Client Services making an effort to improve policies and procedures. Not sure
about contracts and other areas.
There was a period of 10 years where no one did almost anything. Policies and
DBS Management Review
OIG 11/12-20 M
procedures are a work in progress....not where they need to be.
Not sure DBS follows DFS contracting rules.
16)
If you could, what would you change at DBS?
Change leadership and reduce turnover of management ranks.
contributing to communication and organizational issues.
Lots of drama for the size of the Division.
17)
Are there any issues you would like to bring up that we have not discussed?
David mentioned that staff from the Governor's Office came over to meet with the
contracting section. Rumor mill is flying between the OIG project and Governor's
Office visit.
Concerns with staff that have received raises, staff credentials (college degree?),
and how some positions are not advertised just pop up.
Concerns with hiring process enablers paperwork signed off by Linda Champion
and Sheila White.
Turnover
DBS Management Review
OIG 11/12-20 M
Person(s) Interviewed/Title:
Date: 3/9/12
1)
DBS position/ role?
Interviewer: JM
Notes prepared by: KK
Contracts Supervisor, over 3 contract managers and 1 administrative staff.
2)
How long have you worked at DBS?
Since June 2011, approx 9 months. Came from DOE contract administration (there
from 2006 to 6/11).
3)
How long have you worked in your current position? Who is your supervisor?
Same, approx 9 months. Reports to Ellen McCarron.
4)
How would you describe the working environment within DBS/ your office?
Very different. Micro-managed a lot.
5)
Can you name any positive role models or areas that have improved within the
Division?
Contracts and monitoring is doing well. Three really experienced staff and one that
is getting there. Staff doesn't always use
's knowledge and expertise as they
should in other areas.
6)
What is your impression of the staff turnover that has occurred over recent
years?
Very unusual so much turnover. Not used to it. Worries about it.
Broke her foot/leg and has been working from home. Doesn't want to lose job. On
probation for one year as a new employee.
7)
Do you feel DBS contracts are properly negotiated and managed without
preferential treatment?
Different. Most are exempt (from the traditional procurement process). FAASB
(Florida Association of Agencies Serving the Blind) has a lot of control on upper
DBS Management Review
OIG 11/12-20 M
management. FAASB represents the interests of the CRP's.
Negotiations There are criteria. Joyce makes the decisions.
Doing a lot of price analysis.
8)
Do you feel the needs of DBS clients are being met effectively?
She is learning more about actual services supplied to clients, from reporting.
Some reporting makes you wonder if the CRP's are doing everything to serve
clients. Example, time spent with a client and number of clients served.
There are remedy tables on some contracts, but not all. Not the same terms on
different types of contracts.
QPIS and MIS reporting are getting better.
Have redone the monitoring procedures manual.
9)
Are DBS contract managers and staff allowed/ authorized to effectively manage
contracts with CRPs?
In the past, contractors (CRP's) contacted Joyce if they weren't happy. Joyce says
less of that is happening now. Districts have more contact with CRP's and feel
comfortable working together.
Said that some issues arose when DBS moved from AWARE to AWARE 5.11.
Unreal the number of clients that should have been closed in 90 days, that weren't.
Old cases that should have been closed (from 06/07). Doesn't affect payments to
CRP's.
Concerns with invoices that bill for clients that are documented as absent for the
day billed.
Auditor note:
invoices.
10)
did not seem to have confidence in the accuracy of the
Do you feel that DBS funding is being put to its best use?
As far as she knows.
11)
Is the DBS hiring process fair?
DBS Management Review
OIG 11/12-20 M
Not sure. One contract manager without experience was hired for more money
than one's with experience. Same contract manager told
he could have had
her job.
Auditor Note: Believe
12)
was referring to
. KEK 3/9/12
Do you have any sense of the presence of favoritism in DBS hiring practices?
Appears that way. See example above.
13)
Do you feel DBS employees are free to express themselves without concern?
Doesn't feel free to say all her concerns. Doesn't want to be reprimanded or lose
her job.
14)
How would you describe communication from and with executive management?
Has gotten better lately. Appears trying to get better in last few weeks. Good to
know what's going on and what you need to do next.
15)
Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
Thinks so.
Worked hard on her performance evaluations. Asked to change one performance
evaluation. Evaluation was on in December, but asked to change it in January.
Asked to consider other things that happened before she took her position. Made
her feel uncomfortable.
16)
If you could, what would you change at DBS?
No discrimination. Work as a team.
Was told to watch certain employees told by HR person and Director.
17)
Are there any issues you would like to bring up that we have not discussed?
Different work environment. Partiality. Didn't expect environment to be the way it
is. Others agree.
DBS Management Review
OIG 11/12-20 M
Gave an example of the CRP's unique relationship with DBS. Said
was wrung out by the CRP's about contracts/financial consequences.
DBS Management Review
OIG 11/12-20 M
Person(s) Interviewed/Title:
Notes prepared by: J. Maxwell
1)
DBS position/ role?
Interviewer: J. Maxwell Date: 3/19/2012
Official title is program administrator. Before this, she was a program consultant.
Before this, she was a DBS district administrator for eleven years - Saint
Petersburg district.
2)
How long have you worked at DBS?
32 years
3)
How long have you worked in your current position? Who is your supervisor?
Three years. Report to
4)
How would you describe the working environment within DBS/ your office?
Tense. Stressful. Very little support from upper management, if any.
Turnover significant-staff are wondering who is next.
Fear.
She has heard people say that they should be happy that they do not work in the
State Office. She used to love her job. But during the last two-three years, big
changes have made it difficult. She does not know who to trust. Best to stay off of
the radar and keep your mouth shut.
5)
Can you name any positive role models or areas that have improved within the
Division?
Quality assurance has been given more attention recently. This is the section that
is in charge of. She said that she was never trained for this position, that
she learned on her own but gets little support for her initiatives.
6)
What is your impression of the staff turnover that has occurred over recent
years?
Concerned. Don't recall this level of TO before in such a short term. Staff that are
left do not understand why the TO has occurred. If this was a private business, the
boss would be in trouble. Staff has been forced to resign or be terminated. Staff
has been forced to move, reorganizations have taken place, terminations, position
DBS Management Review
OIG 11/12-20 M
descriptions have been changed. Too much has happened too quickly.
Uncooperative staff is "marked" not wanted and need to get rid of.
7)
Do you feel DBS contracts are properly negotiated and managed without
preferential treatment?
Not at all. Employees of the CRP's are not involved in a healthy partnership with
DBS staff. The DBS staff appear to need the permission of CRPs to do just about
anything. Workgroups that are set up must include representatives from CRPs.
Often, DBS employees must sit back and simply say OK. If something is not
written into code or statute, it is "fine."
Much money is being sent to CRPs for new programs leaving less money available
for clients (direct services). Much money is given to the Tampa Lighthouse (job
placement). The Statler Institute for Food Service received much funding but
resulted in very few placements (1 or 2 still working).
Instead of looking for other vendors, money is always directed to Lighthouses,
whether skilled or not. The Lighthouses have become the "experts" on everything.
8)
Do you feel the needs of DBS clients are being met effectively?
In her current position, she is far removed from DBS clients. But she's concerned
that clients are not being offered appropriate choices for services. Clients are
asked to accept services from the lighthouses without being given any other
choices. Money sent to CRPs reduces money that can be spent on clients through
direct services.
9)
Are DBS contract managers and staff allowed/ authorized to effectively manage
contracts with CRPs?
No. Some CRPs have all over a hundred clients. Monitoring performed by the
State Office is 5 cases a month only. Only look at certain aspects of the process.
Example-service hours make sense? Yes or no. The monitoring process currently
used is not effective.
10)
Do you feel that DBS funding is being put to its best use?
No. More and more money is being sent to CRP contracts. Less money is being
given to the district service budget. Travel funding for workgroups has increased.
CRPs provide job placement training but there are other potential providers which
are not utilized. Example - DVR.
DBS Management Review
OIG 11/12-20 M
11)
Is the DBS hiring process fair?
Not sure. Orlando district administrator retired. A CRP employee was placed there
as a supervisor and later made DA. She felt that he does not have the
qualifications. The name is
.
Team performing interviews for DAs is questionable. Only one of the recently hired
DAs has stayed on (hired from outside of DBS with people who don't have
rehabilitation backgrounds). Seems administration is getting back to people with
appropriate experience recently.
12)
Do you have any sense of the presence of favoritism in DBS hiring practices?
Yes. Program Manager
- not qualify for a position that supervises all
district administrators. This does not seem appropriate.
13)
Do you feel DBS employees are free to express themselves without concern?
No. DBS employees are now gone due to speaking up. Does not feel she can
voice thoughts/ opinions which out it being held against her. Fear of retaliation.
14)
How would you describe communication from and with senior management?
Non-existent. Told "here is what I need you to do". Phone calls and emails can go
for long times without being answered or not answered at all. Not a two way
communication that is useful.
15)
Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
To a point. Updated to a degree. However, not enough training given to staff.
Need more support, training. Especially need training for the districts in the area of
personnel issues and processes.
16)
If you could, what would you change at DBS?
The working environment. Voicing thoughts and being recognized for contributions.
Job not in peril if you do not agree with the executive director. Change how we
work with clients. The current order of importance by the executive director is first
the CRPs, then certain staff at the State Office, then clients and finally DBS staff.
DBS Management Review
OIG 11/12-20 M
It would be a good idea to have other vendors provide services and not limit all
contracting with the existing CRP's. Best if you work in an environment where you
do not have fear when you come to work. Too many reorganizations. DBS
counselors need to be respected for what they can contribute. They are currently
not utilizing their skills. Instead they're pushing paper. They should be in involved
more with clients.
17)
Are there any issues you would like to bring up that we have not discussed?
Do not get support from the executive director. Employees are required to change
their views or get out. Too many changes. No one can breathe. If people cannot
adjust to the executive director's methods, they're considered a liability.
Mentioned an episode involving a meeting with the client services Bureau chief.
She was put through a list of questions from employees who "did not like her". It
turns out that some of the employees had never worked for her. The process was
a hoax. It appears to have been orchestrated to get a termination. No proof to the
allegations.
agreed at the end, but said that she was directed
to do this (likely directed by her boss). This was very humiliating for
.
Many employees have been mistreated (ex.
). More money to CRPs
when it could be used for staff raises or direct services to clients.
DBS Management Review
OIG 11/12-20 M
Person(s) Interviewed/Title:
Date: 3/9/12
1)
DBS position/ role?
Interviewer: JM
Notes prepared by: KK
Contract manager for 24-26 contracts in the South Florida region (south of highway
60).
2)
How long have you worked at DBS?
Since 1985, 27 years.
3)
How long have you worked in your current position? Who is your supervisor?
1.5 years. Reports to
4)
.
How would you describe the working environment within DBS/ your office?
Hostile, uncomfortable, afraid to express yourself. Various people feel picked on.
Seen a lot of turnover.
5)
Can you name any positive role models or areas that have improved within the
Division?
Role models no.
Contracts section has attempted to make improvements in the structure of
contracts.
6)
What is your impression of the staff turnover that has occurred over recent
years?
Division has had a lot of turnover. People leaving frustrated and in disgust. Leads
to hostile working environment. Staff replaced with people with little or no
experience with blind services. Experience in the field is not validated. Need a mix
of new and experienced staff. Recruitment is a challenge.
7)
Do you feel DBS contracts are properly negotiated and managed without
preferential treatment?
Not negotiated. Based on number served, historical. Districts have little or no
DBS Management Review
OIG 11/12-20 M
involvement in contracting/amount.
One deliverable - # served. Dollar amount based on figure done by an outside
group.
8)
Do you feel the needs of DBS clients are being met effectively?
Doesn't know. Has yet to see any kind of satisfaction survey results.
Based on units of service, would say no. Should focus on units of service per client
vs. number of clients served.
9)
Are DBS contract managers and staff allowed/ authorized to effectively manage
contracts with CRPs?
Neutral. He's more aggressive and not concerned about negative feedback.
Us vs. them mentality. Districts encouraged not to interfere with CRP's
happening more recently. There is one point of entry, the Division Director. CRP's
call Division Director, not contract managers or District Administrators if they have a
problem. It has evolved to this over a period of time.
10)
Do you feel that DBS funding is being put to its best use?
No, in his personal opinion. Gave Palm Beach as an example. Palm Beach CRP
went bankrupt and they did not have another provider in that area. The District
office took over the independent living services. Were able to do that due to good
staff and networking. District office was able to provide services better and for less.
Services should be a combination of CRP and district services.
11)
Is the DBS hiring process fair?
Doesn't know. Doesn't know where they recruit from. A lot of people hired without
blind experience.
12)
Do you have any sense of the presence of favoritism in DBS hiring practices?
Yes, couple of instances.
13)
Do you feel DBS employees are free to express themselves without concern?
DBS Management Review
OIG 11/12-20 M
No. Really uncomfortable feeling.
14)
How would you describe communication from and with executive management?
Tell them what they want to hear.
Hard to throw out new ideas or be philosophical.
Try not to engage in any kind of confrontation.
15)
Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
Weak, very general. With general it is hard to measure outcome. Independent
language should cite a specific goal.
16)
If you could, what would you change at DBS?
Easy fix new leadership.
Attitude issue with the staff. Hard to change.
17)
Are there any issues you would like to bring up that we have not discussed?
Perception of staff. A coworker has been unfairly harassed by other staff.
Uncomfortable with it. Cliques, groups. A lot of good people - don't like to see
people picked on.
Focus group example: a CRP staff said "what is she doing here" to Joyce Hildreth
in a focus group meeting. Joyce told the DBS staff member (the "she") to leave.
Management doesn't stand up to CRP's for the staff.
He has been watching staff members being moved around based on (rumored)
comments from CRP's.
Seen results of what happens when people pursue thoughts contrary to
management.
Little interaction from the Districts. They won't express themselves. Should want to
hear from the Districts to know when something is coming.
DBS Management Review
OIG 11/12-20 M
Not a lot of initiative or creative thought because everything is micro-managed.
Said his supervisor was "neutered".
Haven't done any kind of monitoring in 2 years. Previous supervisor (
) was fired when he tried to monitor. Dialogue to do it, but nothing
happens.
DBS Management Review
OIG 11/12-20 M
Person(s) Interviewed/Title:
Date: 3/5/12 Notes prepared by: KK
1)
DBS position/ role?
MIS Manager. Manages IT computer support staff and DBS applications.
2)
How long have you worked at DBS?
2 years in April.
3)
How long have you worked in your current position?
2 years in April. Reports to Director,
4)
.
Interviewer: JM
How would you describe the working environment within DBS/ your office?
(Examples, reasons, further information....)
Overall, enjoys it. Especially working with customers.
Described the management team as working with a volcano. Some staff talk to
Joyce a lot and some do not. Information/views get distorted and you don't always
get to defend yourself. Doesn't feel good. There are different "camps" in the office.
Joyce tends to believe the information from the last person she talked to. Hearsay
sometimes not objective.
Gave an example of working on federal reports. Some staff overreact to little
issues and blame MIS.
feels MIS corrects their issues; however, incorrect
data from business areas is what causes the issues, not MIS.
looks at
improving the process rather than all the fixes.
5)
Can you name any positive role models within the Division?
No role models named, but he feels that everything has improved. They are doing
a lot of good things, but doesn't always feel like it. Feels like there is a time bomb
sitting out there. Feels like he can do 10 good things, but 1 instance of a bad thing
and he could be gone. Decisions to terminate an employee are not always
objective. He referred to
. Feels that
helped turn
things around and his departure was unjustified. Some staff are very sensitive and
have a hard time being challenged.
termination may be
questionable too.
*Feels that Joyce ties him to
and that could be seen as a strike
DBS Management Review
OIG 11/12-20 M
against him.
6)
What is your impression of the staff turnover that has occurred over recent
years?
Some good, more good than bad. A lot of good turnover at the District
Administrator level. Digging out from 20 years of bad management.
7)
Do you feel DBS contracts are properly negotiated and managed (no preferential
treatment)?
Not close enough to this area to have an opinion about it.
8)
Do you feel the needs of DBS clients are being met effectively?
Not close enough to this area to have an opinion about it.
9)
Are DBS contract managers and staff allowed/ authorized to effectively manage
contracts with CRPs?
Not close enough to this area to have an opinion about it.
10)
Do you have a sense that DBS funding is being put to its best use?
Not always. Gave an example regarding the AWARE system and Alliance
(contracted support). Feels additional IT services from Alliance (requested by DBS
staff) are not worth the expense.
Been called an "obstructionist" and told he doesn't get to make that decision.
Said Joyce thinks he puts his hand in the business side of things too much. Joyce
said MIS should be a servant to operations/ client services.
11)
Is the DBS hiring process fair?
Doesn't know. Some people (not him at this time) don't think it is open enough.
Happy with his two employee appointments.
12)
Do you have any sense of favoritism in hiring practices?
DBS Management Review
OIG 11/12-20 M
Said it is normal used to people getting people they know.
13)
Do you feel DBS employees are free to express themselves without fear?
No (emphatic). Verbally promotes openness, but actions don't show that.
He has learned to pick his battles. Sit back and watch bad decisions being made.
14)
How would you describe communication from and with executive management?
Effective?
Troubling. Good in some cases. For example, good when all senior managers are
in a room together, but when they break up in "camps" it stirs up drama. He is
busy, doesn't see too much of it, but gets wind of it. Communications are not
clear/fair. He is concerned that he is not involved in some conversations that
involve him.
Jim asked
if he would give the names of the "in-camp".
named
.
gives Joyce, but she is a confidant.
Doesn't know what kind of advice
15)
Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
Thinks so; For the most part
16)
If you could, what would you change at DBS?
Couple managers in over their heads, not strong enough to lead and not great
decision makers.
Staff issues should be dealt with at that level and not by running to the Director.
Would like to be judged based on work performance rather than hearsay.
Have to get rid of the drama mode. Get executive management to work together all
the time. Get rid of "camps".
17)
Are there any issues you would like to bring up that we have not touched upon?
He is not the only one that feels this way. Not the only one that deserves to be
here and shouldn't have to feel this way. On pins and needles. Feels stronger
DBS Management Review
OIG 11/12-20 M
leaders are put in "other camp". Joyce does not seek drama, but drawn into it by
certain staff.
Not making sound decisions. Example, management wanted to be able to run a
report of how many authorizations are placed after the fact. Would be a $12,000
system change through the software provider. Feels you still need to do spot
checks, don't need the system report. Already know where problems are. Doesn't
need system change. Recommended process change create an emergency
authorization process that would allow a one day turnaround (potential best
practice to consider).
Also expressed concern with
going to Joyce over issues she
has with him (
). They work together a lot. She doesn't like controversy.
Would rather she come to him with her problems about him.
DBS Management Review
OIG 11/12-20 M
Person(s) Interviewed/Title:
Date: 3/22/12
1)
DBS position/ role?
Interviewer: KK
Notes prepared by: KK
Rehabilitation supervisor, District 6 - Orlando
2)
How long have you worked at DBS?
18-19 years
3)
How long have you worked in your current position? Who is your supervisor?
5.5-6 years total.
Reports to
4)
Took a year off to work as a counselor to help her district.
.
How would you describe the working environment within DBS/ your office?
In her office fine. Calm, pleasant environment. Recently moved offices so all
staff are on the same floor.
Overall A lot of stress. Caseloads large. Huge influx of newly blind in the area.
5)
Can you name any positive role models or areas that have improved within the
Division?
A lot has improved. For example, the case management system. Now they have
quarterly reviews with counselors to go over how things are going.
has made a huge difference in the office, for the good.
6)
What is your impression of the staff turnover that has occurred over recent
years?
Stresses her terrifically. Lose continuity with turnover. Terrible turnover in districts.
Need good people to come and stay. No senior counselors at this time. Most staff
are kind of new.
7)
Do you feel DBS contracts are properly negotiated and managed without
preferential treatment?
DBS Management Review
OIG 11/12-20 M
Thinks so. A lot of tension with facilities. Not due to negotiations. A lot of tension.
(LCF and Lake County)
8)
Do you feel the needs of DBS clients are being met effectively?
As much as they are capable of. Experience a gap in service due to turnover
vacant caseloads. Affects clients served. Affects speed of service.
9)
Are DBS contract managers and staff allowed/ authorized to effectively manage
contracts with CRPs?
Yes.
10)
helps and is supportive.
Do you feel that DBS funding is being put to its best use?
Yes, at the district they do their best.
We discussed the pilot job placement program with LCF. Lighthouse received
$4,000 for a placement. Have 40 people in the program. Have placed 9, 6 of
which have been in a LCF facility. A little uncomfortable with the numbers. Wants
to involve other businesses. Stress over authorizations. Money given for training,
money given for placements, then put back in for training. Counselors have their
own goals for placements. If the goals are not reached they will receive
unsatisfactory evaluations. Lighthouse ties up placements. Lighthouse decides
when someone is ready for a placement. District has no control over their numbers
and working on this pilot program causes them to lose time for their own job
placements (which have to be in by 3/30).
11)
Is the DBS hiring process fair?
Yes, in her office. Pick someone with the appropriate skills and education.
12)
Do you have any sense of the presence of favoritism in DBS hiring practices?
No.
13)
Do you feel DBS employees are free to express themselves without concern?
Personally, she is free to speak. Doesn't know if all her staff would say the same,
because they are new.
DBS Management Review
OIG 11/12-20 M
She and
14)
have team meetings. Staff are encouraged to be open.
How would you describe communication from and with senior management?
Doesn't have much contact. Gets along fine when she sees them. Communication
is through
.
15)
Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
has been working hard on policies.
policies/expanded on information.
Policies are the best they've been in a long time.
16)
If you could, what would you change at DBS?
Give staff more positive feedback. There is a focus on what people are doing
wrong auditing perspective.
Feels that she has lost a lot of people because they don't feel like they were
appreciated.
17)
Are there any issues you would like to bring up that we have not discussed?
Worried about the LCF situation. It is a big program, lots of clients involved, and
has grown dramatically. Grown so fast. Need to work to make it a team effort.
Need teamwork.
Has cleared up gray areas in
DBS Management Review
OIG 11/12-20 M
Person Interviewed/Title:
Date: March 9, 2012
, former Deputy Director, DBS
Interviewer: Dean Goodson
1) Former DBS position/role?
Deputy Director, DBS.
2) How long did you work at DBS?
Under two years, August 2009 to April 2011.
3) How long did you work in your current position? Who was your supervisor?
One year and Joyce Hildreth, DBS Director was his supervisor.
4) When were you terminated or did you leave DBS?
stated that he was given the option to resign in lieu termination. He said
that Hildreth gave him this option in the presence of
in the Personnel
office.
5) Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
explained that policies in about half of the areas are very good, but in other
areas, policies are lacking, vague, or do not exist. He said this was in areas of
strategic planning, projects with management, and priority setting by management.
said the management information systems and contract management
were examples of this; and personnel and clients services had good policies and
procedures.
6) How would you describe the working environment within DBS/ your office?
said in DBS, the intent was there, but including himself, other employees
were fearful of losing their positions. He said employees were afraid of expressing
their thoughts or ideas.
7) Can you name any positive role models or areas that have improved within
the Division?
said the overall view of DBS by their stakeholder groups improved
because of more communication and more listening by DBS which created a
better relationships. He said he especially saw this with Business Enterprise and
the management information systems.
DBS Management Review
OIG 11/12-20 M
8) What is your impression of the staff turnover that has occurred over recent
years?
said he had a "mixed impression." He said the old leadership was
not strong with holding employees accountable. The new leadership, however,
holds employees accountable, and the solution is letting employees go when they
find them not accountable, resulting in several employees losing their jobs.
said conflicts were dealt with through personnel actions such as
terminations rather than through more constructive approaches to solving an issue
or problem.
9) Do you feel DBS contracts are properly negotiated and managed without
preferential treatment?
said there are "gaps" in the DBS contracts. Some are managed
consistently while other are not. He said there needs to be more competitive
bidding especially with client services, aids, or technology. There needs to
be more scrutiny with the administration of the contracts.
10) Do you feel the needs of DBS clients are being met effectively?
said overall the needs were being met because the bulk of funding was
dedicated to vocational rehabilitation.
11) Are DBS contract managers and staff allowed/ authorized to effectively
manage contracts with Community Rehabilitation Providers (CRPs)?
explained there was a lot of micromanaging of the contract decisions by
Hildreth, and contract managers tried to manage them, but were timid to exercise
the terms and conditions of the contracts for fear of repercussions. They did not
want to go against what Hildreth wanted, and what was written in the contract was
not always enforced.
12)
Do you have a sense that DBS funding is being put to its best use (e.g.,
increased dollars in CRP contracts)?
said that the authorization process of the vocational rehabilitation
program, where a fourth to a third of the funding goes, should be reviewed to
determine if it is a sound process. He said the process should be internal to
DBS without going through upper management review or anyone outside of DBS.
13) Is the DBS hiring process fair?
said "yes overall" at the staff level. But, at the management level, hiring
became whatever route was expeditious instead of taking the time to acquire the
best candidate for the position.
DBS Management Review
OIG 11/12-20 M
14) Do you have any sense of the presence of favoritism in DBS hiring
practices?
responded that he did not have a sense of favoritism, but it was more of
filling a position too quickly with the management level positions. There were no
value or consistency with placing someone in those positions. The attitude was
"we'll just get another one," and not necessarily a qualified person.
15) Do you feel DBS employees are free to express themselves without
concern?
said the intent was genuinely there, but in practice, it was not that way.
16) How would you describe communication from and with executive
management?
said communication was inconsistent. He said there would be "a load for
awhile then there would be none for awhile."
17) If you could, what would you change at DBS?
said top priorities are not clear or concise to employees, there are too
many snap decisions from management causing employees not to understand
what is important. He said if there was better direction from the top, the turnover
at DBS would stop.
18) Are there any issues you would like to bring up that we have not touched
upon?
said just the way things change so quickly at DBS. He said two and a
half weeks before he was told to resign, Hildreth told him "we make a great team
together."
DBS Management Review
OIG 11/12-20 M
Person Interviewed/Title:
Technology Consultant, DBS
Date: March 13, 2012
, former Rehabilitation Engineer and
Interviewer: Dean Goodson
1) Former DBS position/ role?
Several positions: Rehabilitation Engineer and Technology Consultant, Program
Manager of Client Services, Bureau Chief of Client Services, and Bureau Chief of
Business Enterprises.
2) How long did you work at DBS?
16 years.
3) How long did you work in your current position? Who was your supervisor?
March 2009 to November 30, 2011.
4) When were you terminated or did you leave DBS?
stated he was told he had to move to Daytona, FL or be terminated. He said
he elected to resign in lieu of termination because he did not want to make the
move.
said he learned that after he resigned, the position he was to take in
Daytona was eliminated.
5) Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
said he believed they are when they are followed. However, DBS staff did not
follow the policies in order "to keep the Director from coming down on them." He
gave examples of two of the Lighthouses for the Blind where the administrators
reported to the DBS Director, Joyce Hildreth, that these CRPs were not servicing
the clients properly and not documenting the services correctly.
said Hildreth
got upset with the administrators and told them they should be "working with them"
rather than providing the right services or documentation.
6) How would you describe the working environment within DBS/ your office?
said the environment was hostile and employees called Hildreth "Hitler"
because she ruled by intimidation. He said she would select an employee time
after time and give them a hard time.
7) Can you name any positive role models or areas that have improved within
DBS Management Review
OIG 11/12-20 M
the Division?
said the relationship with the Community Rehabilitation Providers. But, he
said it was a "reverse type of relationship" because DBS worked for the CPRs
instead of the CPRs working for DBS.
8) What is your impression of the staff turnover that has occurred over recent
years?
said this was the highest turnover DBS ever had. He said several employees
left just to get away from DBS while several others were wrongfully terminated.
9) Do you feel DBS contracts are properly negotiated and managed without
preferential treatment?
said not with the CRP contracts. He said funding was taken from direct
services and placed in CRP contracts especially with training, which meant clients
had no other choice for training except through the CRPs. He said as more
money is transferred to the CRPs, there is less money dedicated for other client
direct services. He added that the CRPs have now received the contracts for job
placement and they do not have the proper training that the DBS counselors have.
10) Do you feel the needs of DBS clients are being met effectively?
said "no" based upon his previous answer and that clients cannot receive the
tools and devices they need from direct services because funding is going to the
CRPs.
11) Are DBS contract managers and staff allowed/ authorized to effectively
manage contracts with Community Rehabilitation Providers (CRPs)?
said "no." He gave the example of CRPs not wanting
as a
contract manager because he spoke out against CRPs receiving funds when they
did not meet their numbers and obligations to clients. Instead,
was
transferred out of his position to the Talking Book Library.
12) Do you have a sense that DBS funding is being put to its best use (e.g.,
increased dollars in CRP contracts)?
responded "no." He gave the example that two years ago, DBS received
stimulus funds, which was used for a grant to recruit blind students at colleges for
a job opportunity, when in fact DBS already had knowledge of all of these students.
He said they wasted these funds anyway for the grant. He added that DBS spent
money for CRPs to update their equipment technology when they were supposed
to already have this equipment to perform under their contracts. Elliott said this
money should have gone to client direct services.
DBS Management Review
OIG 11/12-20 M
13) Is the DBS hiring process fair?
said "no," that employees who were more than qualified were not promoted
or selected for better positions because Hildreth or some other upper level
manager did not like them.
14) Do you have any sense of the presence of favoritism in DBS hiring
practices?
said "yes," that there were individuals selected for supervisory positions that
lacked experience or were not qualified; but because they were liked by Hildreth or
by someone in upper management, they were hired.
15) Do you feel DBS employees are free to express themselves without
concern?
said "no, definitely not," that staff members are afraid to express their
opinions especially about the wrongful practices associated with the CRP
contracts. He added that probably around 80 percent of the DBS staff are
afraid to come to work.
16) How would you describe communication from and with executive
management?
explained that it was mostly monologues by Hildreth rather than dialogues
with DBS staff members. He said employees and upper management are afraid
of losing their jobs if they cross Hildreth in any way.
17) If you could, what would you change at DBS?
said he would change "the atmosphere" with the way employees are treated,
and change the way business is conducted with the CRP contracts by changing
the funding formulas and procedures.
18) Are there any issues you would like to bring up that we have not touched
upon?
recommended that someone needs to take a close look at all the DBS
contracts because their number continues to increase when utilizing other
competitive client services would be cheaper and beneficial to their customers.
He further stated that the way
, DBS Personnel Liaison, conducts her
business is "not on the up and up." He described White as "Hildreth's henchman."
OIG 11/12-20 M: DBS Management Review
Person(s) Interviewed/Title:
Date: 3/2/12 Notes prepared by: Kelly Kilker
Interviewer: Jim Maxwell
1)
DBS position/ role?
Bureau Chief of the Bureau of Business Enterprise
2)
How long have you worked at DBS?
7 years
3)
How long have you worked in your current position?
1.5 years. Supervisor is Joyce Hildreth (Division Director).
4)
Are written policies and procedures adequate to guide staff in core functions?
And, are they following and enforced?
Yes and yes.
5)
How would you describe the working environment within DBS/ your office?
(Examples, reasons, further information....)
Extremely good. Climate has changed. Believes there has been an upgrade of
professionalism and accountability has changed. Joyce Hildreth wanted a year to
evaluate staff. She said everyone would be held accountable for their position
description (which
said was needed). Many people left due to being held
accountable.
said that before Joyce: 1) Providers had no respect for DBS; 2)
He only had respect for the Ft. Lauderdale district office; and 3) they had to have
work groups to address the "wishy washy" contracts.
believes there is a greater
respect for DBS now.
6)
Can you name any positive role models or areas that have improved within the
Division?
named Joyce first.
He also proceeded to list several of the District
Administrators (DA) and other staff: Ivy Romero (Ft Lauderdale DA); Tony Pileggi
(Pensacola DA); Michelle Levy (Miami DA); Brian Michaels (Orlando DA); Jeffrey
Whitehead (Tampa DA); and Jim Woolyhand (Daytona Beach DA).
said that he
has respect for the DA's that he knows. He said he doesn't know if Ana St. Fort
OIG 11/12-20 M: DBS Management Review
(Tallahassee DA) is fit for the job. The districts have received new supervisors and
DA's. A lot have been brought from the outside.
The relationship between districts and the providers has improved.
Policies and procedures are followed more consistently regarding any issue the
previous Director used to operate based on mood/feelings.
7)
What is your impression of the staff turnover that has occurred over recent
years?
Accountability thing some couldn't adjust to the changes.
discussed the Bureau of Administrative Services. Lots of change. Probably
warranted.
left and
took the position.
couldn't
explain things (e.g. the budget). She is no longer in the position.
took over. He couldn't explain the budget either. Now
is in the
position.
believes she is a good fit.
also discussed the Deputy Director position.
was nice but
didn't do anything.
took over the position.
was a good guy,
but busy - trying to make up for the lack of work from his predecessor. Too much.
was below
and they clashed.
was shocked when
was let go. It was sudden. Staff were interviewed and several felt threatened by
. Ellen took over the position.
likes
in some ways. Said she is
smart, but gets confused. Doesn't think she was the right fit for the position. Not
good people skills. Doesn't handle certain things well.
8)
Do you feel DBS contracts are properly negotiated and managed (no preferential
treatment)?
has been away from contracts for the last 18 months. Contracts are far better
than before Joyce came. Steve Ritacco would do what he wanted to do with
contracts. Now, things have been put in place to make contracting fairly uniformed.
Amounts contracted based on historical data/trends.
Does not know of recent overpayments.
Could have been tougher on some of the smaller/struggling providers, but didn't
want to lose provider (some providers depend on DBS funds to stay in business).
Does not know or does not feel that there has been preferential treatment. Larger
providers receive more funds because they see more clients. Money is tied to
people served.
OIG 11/12-20 M: DBS Management Review
*
pulled
aside one day to ask him about the contracts.
said the
Department of Financial Services wanted to know why DBS contracts have a
remedy asking a provider to pay back funds. Contracts used to have a remedy that
the last payment would be withheld or reduced.
is not sure about new
language.
Follow up: Audit staff will review contracts to see if remedies language has
changed.
9)
Do you feel the needs of DBS clients are being met effectively?
Yes, but doesn't work closely with Client Services.
personnel are better.
Would think yes since
referenced Florida Statute that calls for services to be contracted as much as
possible. Believe
was referring to: §413.014, F.S., which directs DBS to "as
rapidly as feasible, increase the amount of such services provided by community
rehabilitation programs."
District's role has diminished, and they are not happy. Resentment. Less people in
districts. Many district staff have been hired by the providers. DBS expects more
from the providers and less from DBS staff.
Jim asked what
thought about a comment made that more contracted money
meant less for direct services.
initially had no comment, but referred to
resentment from district offices.
10)
Are DBS contract managers and staff allowed/ authorized to effectively manage
contracts with CRPs?
No idea at this point/no opinion.
11)
Do you have a sense that DBS funding is being put to its best use (e.g.,
increased dollars in CRP contracts)?
Yes, in his area (Bureau of Business Enterprise).
12)
Is the DBS hiring process fair?
Yes, as far as he knows.
13)
Do you have any sense of favoritism in hiring practices?
OIG 11/12-20 M: DBS Management Review
No, no evidence.
14)
Do you feel DBS employees are free to express themselves without fear?
Yes.
15)
How would you describe communication from and with executive management?
Effective?
Yes. Meetings and emails all the time.
16)
Are there any issues you would like to bring up that we have not touched upon?
Overall, Joyce has done an excellent job.
pointed to hiring concerns from Craig Kiser. He said Craig believed that a blind
person can do any job a sighted person can do. Craig placed a lot of blind
individuals into jobs they weren't necessarily qualified for (e.g. management
experience/skills).
17)
If you could, what would you change at DBS?
Discussed concerns/contention with DBS Fiscal staff on the 9th floor (
and
). DBS used to waste money, but they are more fiscally
responsible now and trying to do all the right things. Discussed concerns with
communications issues between fiscal staff and other DBS staff. Gave an example
of fiscal staff contacting Joyce and a meeting being held when the problem could
have been resolved by a phone call.
DBS Management Review
OIG 11/12-20 M
Person(s) Interviewed/Title:
prepared by: GW
1)
DBS position/ role?
District 3 administrator
2)
Interviewer: GW Date: 3-12-2012 Notes
How long have you worked at DBS?
Since October 2011.
3)
How long have you worked in your current position? Who is your supervisor?
Since January 2012.
4)
How would you describe the working environment within DBS/ your office?
Supportive. As a new employee, I think I can go to my supervisor for answers and
get good information. Very professional. Well managed.
5)
Can you name any positive role models or areas that have improved within the
Division?
. She started as a counselor and moved up. Good model for me.
6)
What is your impression of the staff turnover that has occurred over recent
years?
Has been turnover in the Jacksonville office. Reasons vary and are the same as
other places I have been: Low pay, personal reasons, and leadership challenges.
7)
Do you feel DBS contracts are properly negotiated and managed without
preferential treatment?
Yes, I review contracts monthly. Use a checklist. Don't see preferential treatment,
the numbers are either there or they're not.
8)
Do you feel the needs of DBS clients are being met effectively?
Yes. Appropriate policies and procedures are in place.
DBS Management Review
OIG 11/12-20 M
9)
Are DBS contract managers and staff allowed/ authorized to effectively manage
contracts with CRPs?
Out of my purview. I know who to contact in Tallahassee.
relationship with my CRPs. Open and honest relationship.
10)
Do you feel that DBS funding is being put to its best use?
Overall, yes. Funding is there for the VR program. Some other programs could
use additional funding: blind babies, older blind, and Adult Program.
11)
Is the DBS hiring process fair?
I believe so. Sometimes the decision to hire takes too long. Lot of hands involved.
12)
Do you have any sense of the presence of favoritism in DBS hiring practices?
No.
13)
Do you feel DBS employees are free to express themselves without concern?
Yes. The way things are expressed needs to be constructive, professional. Don't
have problems going to my supervisor. She is timely in responding.
14)
How would you describe communication from and with senior management?
Haven't had much communication. Just e-mail. I believe in the chain of command.
My experience has been super.
15)
Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
Adequacy depends on which policy and procedure. Some policies are a little too
free and favor the client too much. Most are good with good balance between DBS
and the client. I'm still too new to the job to really respond.
16)
If you could, what would you change at DBS?
New counselor training would be helpful at the beginning of employment. We have
continuing training but training specific to new counselors would be helpful.
I have a positive
DBS Management Review
OIG 11/12-20 M
17)
Are there any issues you would like to bring up that we have not discussed?
No.
DBS Management Review
OIG 11/12-20 M
Person(s) Interviewed/Title: Ana Saint-Fort Interviewer: GW Date: 3-12-12 Notes
prepared by: GW
1)
DBS position/ role?
District two administrator.
2)
How long have you worked at DBS?
18 years.
3)
How long have you worked in your current position? Who is your supervisor?
6 years.
4)
How would you describe the working environment within DBS/ your office?
Average.
5)
Can you name any positive role models or areas that have improved within the
Division?
Within the last two years communication has improved.
good role model.
6)
What is your impression of the staff turnover that has occurred over recent
years?
In the field, a lot of the turnover has to do with low salaries.
7)
Do you feel DBS contracts are properly negotiated and managed without
preferential treatment?
Need to look at how they are managed. I think it should be on a fee for service
basis. Should depend on the number of clients served not lump sum. Can't
comment on preferential treatment (not in a position to know). More competition
could be brought into the process if it were open to TBI (teachers of visually
impaired) individuals.
is a
DBS Management Review
OIG 11/12-20 M
8)
Do you feel the needs of DBS clients are being met effectively?
Overall, the Division is striving for that. There are pros and cons with individual
CRPs depends on the needs of the individual client.
9)
Are DBS contract managers and staff allowed/ authorized to effectively manage
contracts with CRPs?
District administrators used to have a more management/monitoring role with
respect to the CRPs. Now that is more centralized. I do look at invoices submitted
for payment.
10)
Do you feel that DBS funding is being put to its best use?
Yes. We try to be good stewards money goes to helping clients.
11)
Is the DBS hiring process fair?
Yes, at the district level. Follow the People First protocol. Can't answer at the
senior management level.
12)
Do you have any sense of the presence of favoritism in DBS hiring practices?
Not at the district.
13)
Do you feel DBS employees are free to express themselves without concern?
Think we are. The Division has been working on improving communications. I feel
free to bring issues up the chain to
.
14)
How would you describe communication from and with senior management?
On a scale of 1(poor) to 5(great), I would say a 4. Pretty good. The Division has
been striving to improve communications.
15)
Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
Policies and procedures are adequate at the field level and are enforced.
protocol is in place for changes and exceptions.
A
DBS Management Review
OIG 11/12-20 M
16)
If you could, what would you change at DBS?
Increasing the amount and quality of communications. Would like for senior and
middle management to meet face to face to talk about trends and best practices.
17)
Are there any issues you would like to bring up that we have not discussed?
Pay is a big issue in order to retain staff.
DBS Management Review
OIG 11/12-20 M
Person(s) Interviewed/Title:
prepared by: GW
1)
DBS position/ role?
District 7 administrator. Tampa
2)
How long have you worked at DBS?
Over 2.5 years.
3)
Interviewer: GW Date: 3/13/12 Notes
How long have you worked in your current position? Who is your supervisor?
Since July 2011.
.
4)
How would you describe the working environment within DBS/ your office?
DBS seems to be more structured then when I started in the St. Pete office. More
down to business. More structured. More reports. More workload. Morale has
suffered due to more workload and no raises. We had to close the St. Pete office
and moved the Lakeland office. Had lots of turnover and a rough transition period
following some personnel/management problems. Things have improved. New
policies implemented. Procedures streamlined. Working with CRPs to improve
relations damaged by the previous district administration.
5)
Can you name any positive role models or areas that have improved within the
Division?
and the BEP program has improved.
are all good role models. The fiscal unit is good.
6)
What is your impression of the staff turnover that has occurred over recent
years?
Seems like a lot of staff turnover in the district. Takes a lot of time and paperwork.
Turnover at state office has caused problems with changing roles and keeping up
with the changes.
7)
Do you feel DBS contracts are properly negotiated and managed without
preferential treatment?
DBS Management Review
OIG 11/12-20 M
I am on the contract team. Contracts are not bid perhaps because providers are
limited. Other agencies I've been with bid more.
8)
Do you feel the needs of DBS clients are being met effectively?
In most cases they are. The level of documentation is high. Maybe if we reduced
the amount of documentation we could give better service. Some Lighthouses are
better than others working on that.
9)
Are DBS contract managers and staff allowed/ authorized to effectively manage
contracts with CRPs?
I work with
. He seems to be effective. Does a good job. If we have a
question, we get an answer.
10)
Do you feel that DBS funding is being put to its best use?
Question the way allocations to districts are made. I don't know how the dollars per
client compares to other states. Funding should be allocated per client not a lump
sum to the CRP. I worked in DVR and they pay for services rendered.
11)
Is the DBS hiring process fair?
Yes for the district. Can't speak for the state office.
12)
Do you have any sense of the presence of favoritism in DBS hiring practices?
No for the district. Can't speak for the state office.
13)
Do you feel DBS employees are free to express themselves without concern?
I haven't had any problems but others have said they don't feel they can speak up.
14)
How would you describe communication from and with senior management?
There are various levels of communication. Communication was good when I
started in 2009. In March 2010 communication seemed to slack off and that
continued to the fall of 2011. Now things have picked up and communications have
improved.
DBS Management Review
OIG 11/12-20 M
15)
Are written policies and procedures adequate to guide staff? And, are they
followed and enforced?
Think they are adequate. Have gotten more. Need to absorb what we have it
has been a bit overwhelming.
16)
If you could, what would you change at DBS?
DVR received across the board raises about two years ago. We need to do the
same. Put a hold on new policies for a while until what we have is solidly in place.
Reduced staff turnover would be good.
17)
Are there any issues you would like to bring up that we have not discussed?
I'm new to the position. Trying to get the district back to where it was before the
problems started. I feel I have the support to make needed changes.
Department of Education
Office of Inspector General
DBS Management Review
Contracting
Interim Results
Client Services Contract Procurement/ Contract Provisions
- Using statutory exemption to contract without competitive bidding (6. Health
services. 7. Services provided to persons with mental or physical disabilities by notfor-profit corporations)
- Using same 15-20 providers each year
- Fixed rate contracts with monthly fixed payment amount
- Primary deliverable/ payment trigger Documented services to a minimum number
of eligible individuals
- Contract award amounts based largely on the actual deliverables of the prior year
Contract Remedies - loosely written and not always enforced
Remedies (penalty or sanction for contract non-performance)
- Language has been softened over last 3 years
o Final numbers served reduced from 100% to 80%. Remedy payment
changed from being taken out during contract term to being billed.
- Per contract (Attachment A):
o No initial check until 6th month. 40% of deliverable (invoiced services)
o Only at 10th month is a potential monetary remedy addressed (Contractors
can serve only 52% of original requirement without penalty)
o Contract completion - 80%-100% clients served is expectation (but no
remedy in contracts)
Examples of non-enforcement of contract remedies:
-
Tampa Lighthouse (contract 11-522 for $336,000) did not meet deliverables. No
remedy applied. Handwritten note on final payment documentation reads "Per
DBS director, contractor was advised they must meet deliverable by May invoice
period."
Lighthouse of Central Florida (contract 10-976 for $700,000) applied remedy by
reducing the last invoice. However, a payment was made from the G&D fund to
reimburse the provider in the exact amount of the remedy.
-
Contract Monitoring
- No monitoring reports issued in last 2 years (no formal monitoring).
- DFS - The monitoring process should include reconciling provider-generated
reports to source documentation such as case management notes, sign-in
sheets and client files.
Contract Close-out (final assessment of contract)
- No written procedures addressing close-out process. After 2008 IG audit, a form
and process were in place (undone?).
- No close-out reports prepared for 2009-10
- Errors noted on 2010-11 close-out reports reviewed
- No reviewer approval on 2010-11 close-out reports
Responses from prior DBS employees on contracting
Agree following have merit:
- Need for competitive bidding for client services contracts, aids and technology.
- Use of limited number of providers for most client services limits client choice/
customer satisfaction.
- DBS staff should have more say about the purchasing of client services.
- More client services are being shifted from DBS to Lighthouses due to current
management strategy.
- The executive director is said to be more involved with negotiations of the
contracts than the actual contract managers.
- The DBS case management system can be more effectively utilized to monitor
services provided to clients.
Review steps
1) Reviewed procedures on contract administration/ contract monitoring
2) Analyzed trends in dollar amount and number of contracts awarded
3) Looked at changes in contract language and remedies for VR contracts over the last
3 years.
4) Inspected contract closeout documentation for the last 2 years for Vocational
Rehabilitation and Blind Babies contracts for a sample of 6 CRPs
- reviewed clients served vs. planned
- noted how numbers of clients served impacted subsequent years award amounts
- determined that contract remedies were imposed and collected (as applicable)
5) Obtained financial information on trends in Division disbursement for contracting
versus authorizations.
6) Met with DFS staff on recent audit that included DBS contract agreements
Department of Financial Services (Preliminary findings)
Our review of twenty-five contracts and grant agreements disclosed that the Department
had scope of work and/or deliverable issues for nineteen of these contracts.
Specifically, we noted the following:
One grant agreement did not contain a scope of work that clearly established the
tasks the recipient was required to perform. The Department provided an
explanation that the scope of work was provided for by statute. However, the
grant agreement did not include this statutory reference.
Contract #
372-96820-1S001
Service Provider
Tallahassee Community College
Contract Amount
$ 316,675
Three service contracts provide for twelve monthly payments; however, there is
not a required level of performance until the end of the sixth month. We
recommend that the Department align the performance with each of the monthly
payments.
Contract #
12-506
12-533
12-526
Service Provider
Conklin Centers for the Blind
Florida Centers for the Blind
Visually Impaired Persons of Southwest FL
Contract Amount
$ 110,331
$ 158,332
$ 233,100
Fifteen grant agreements did not provide for a specific level of service or criteria
to determine successful completion criteria for deliverables. For example,
o The agreement with the University of Central Florida provided for a
consultant to fill the position of Chancellor of public schools for a seven
month period. In turn, the University subcontracted with the consultant. A
general position description was included as an attachment to the grant
agreement. Payments were made to the university quarterly in advance
with no requirements in the grant agreement to provide for a minimum
level of service.
o The two agreements for the Failure Free Reading Program (Washington
County School District and Putnam County School District) did not specify
the minimum number of students to be served.
o Four agreements did require documentation to be submitted with the
invoice to demonstrate performance. However, there were no minimum
performance levels the provider was required to meet when submitting
monthly payments.
Contract #
380-2442A-2CCC1
206-2442A-20001
376-96100-2SB01
130-97260-2SL01
481-60010-1SC01
500-93720-2S001
670-96433-2S001
670-92400-2D001
480-2262A-2CA01
757-4052A-2PFJ1
522-95010-2S001
530-92860-1SA01
37D99650-2QG01
206-93650-1Q001
540-96443-2S001
Service Provider
Levy County School District
Investing In Our Youth
FAMU
Miami-Dade County District Schools
University of Central Florida
Palm Beach County School District
Washington County School District
Washington County School District
Orange County District Schools
Centro Campesino Farmworker, Inc
St. Petersburg College
Polk County School District
Volunteer USA Foundation
Investing In Our Youth
Putnam County School District
Contract Amount
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
89,337
172,998
114,701
175,750
130,000
300,000
375,000
578,156
635,751
94,222
132,995
65,770
200,000
156,402
375,000
Financial Consequences
Effective July 1, 2010, Section 287.058(1)(h), Florida Statutes, requires services
contracts to contain provisions for financial consequences an agency must apply if a
provider fails to perform in accordance with a contract.
The financial consequences for three service agreements allow the vendor to
receive full payment for the contract. If it is determined that financial
consequences should be assessed, the Department will bill the vendor after the
contract is completed. Although this may meet the statutory requirement, the
billing of a contractor for financial consequences after the final payment is made
does not provide a viable solution to encourage contract compliance. In addition,
it is possible the Department may have difficulty in collecting the amount owed.
Contract #
12-506
12-533
12-526
Service Provider
Conklin Centers for the Blind
Florida Centers for the Blind
Visually Impaired Persons of Southwest FL
Contract Amount
$ 110,331
$ 158,332
$ 233,100
Other
The Department was unable to provide a written grant agreement for a recipient of State
Funds. Instead, we were provided the award notification, the recipient's budget of
expenditures and the Request for Award was reviewed. To date, the Department has
disbursed $120,000.00 for this grant award.
Contract #
670-96441-2S001
Service Provider
Washington County School District
Contract Amount
$ 300,000
Contract/Grant Management
Contract/Grant managers must enforce performance of the contract terms and
conditions; review and document all deliverables for which payment is requested by
vendors; and provide written certification of the Department's receipt of goods and
services and ensure all payment requests are certified.
Section 215.971(2), Florida Statute requires services to be accepted in writing
before payment. The grant managers were not certifying receipt of deliverables
for thirteen grant agreements as the payment terms allow the recipients to
receive payments in advance. Regardless of the timing of payments, we are
recommending that the Department maintain signed certification statements in
the management files to support payments on a post audit basis.
Contract #
380-2442A-2CCC1
372-96820-1S001
376-96100-2SB01
130-97260-2SL01
500-93720-2S001
670-96443-2S001
670-92400-2D001
480-2262A-2CA01
522-95010-2S001
530-92860-1SA01
37D99650-2QG01
540-96443-2S001
670-96441-2S001
Service Provider
Levy County School District
Tallahassee Community College
FAMU
Miami-Dade County District Schools
Palm Beach County School District
Washington County School District
Washington County School District
Orange County District Schools
St. Petersburg College
Polk County School District
Volunteer USA Foundation
Putnam County School District
Washington County School District
Contract Amount
$
$
$
$
$
$
$
$
$
$
$
$
$
89,337
316,675
114,701
175,750
300,000
375,000
578,156
635,751
132,995
65,770
200,000
375,000
300,000
The required certification statement for one grant agreement is being signed by
an employee who is not the designated grant manager.
Contract #
481-60010-1SC01
Service Provider
University of Central Florida
Contract Amount
$ 130,000
Deliverables for four agreements were approved based on reconciling the
invoices with data input by the vendor/recipient on the department's reporting
database. The monitoring process should include reconciling provider-generated
reports to source documentation such as case management notes, sign-in
sheets and client files.
Contract #
12-506
12-533
12-526
757-4052A-2PFJ1
Service Provider
Conklin Centers for the Blind
Florida Centers for the Blind
Visually Impaired Persons of Southwest FL
Centro Campesino Farmworker Center, Inc
Contract Amount
$ 110,331
$ 158,332
$ 233,100
$ 94,222
It was noted that monitoring has not been performed for one agreement.
Although by statute FILC is independent of the agency, they are still responsible
for the proper expenditure of funds and use of resources. We recommend the
Department incorporate a monitoring process to include the review of receipts.
Contract #
11-133
Service Provider
Florida Independent Living Council
Contract Amount
$ 395,820
Section 287.057(14), Florida Statute, states that the contract manager
procedures must include monitoring and documenting contract performance,
reviewing and documenting all deliverables for which payment is requested by
vendors. The designated contract managers for two contracts have no contact
with the vendor as their only responsibility is to process the invoices for payment.
The services are validated and approved by other Department staff members. A
similar situation arises with a third contract; however, the contract manager does
have contact with the contractor on issues involving payment of the invoice, but
relies on verbal conversations with another Department staff member for
validation of service. In these instances, these contract managers are certifying
receipt of goods and services with no direct knowledge of the performance of the
contractors.
Contract #
A43012
12-007
A4455E
Service Provider
Creative Consulting
Charles T. Whitelock
Technicsource, Inc
Contract Amount
$ 163,459
$ 160,000
$ 154,252
DEPARTMENT OF EDUCATION
OFFICE OF INSPECTOR GENERAL
INTERNAL AUDIT
Division of Blind Services
Contracted and Purchased Client Services
Final Report
Audit Number
07/08-01A
June 19, 2008
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
Executive Summary
The Division of Blind Services (DBS) provides services to assist individuals, whose primary
disability is visual, in achieving maximum levels of employment, independence, and integration
into the community. These services are provided either directly through DBS district field offices
or indirectly through contracts with community rehabilitation providers. During fiscal year (FY)
2006-07, federal and state funding totaling $11.8 million was spent to purchase products and
services for clients, while $11.1 million was paid for client services provided through contract
providers. Products and services purchased for clients includes, but is not limited to, computer
equipment, software, low vision aids and devices, physical and mental restoration, and training.
Summary of Findings
This audit focused on evaluating the system of internal controls over DBS's procurement and
management processes. This report has findings in each principal activity related to contract
administration including procurement, award agreements, performance, payments processing,
and monitoring. Other findings address the need to improve written procedures to give DBS
employees better guidance when performing key activities, and issues regarding non-client
purchases made using the Accessible Web-based Activity and Reporting Environment
(AWARE) case management system.
Recommendations in the report are intended to assist management in establishing a more
effective control environment that will better ensure procurement and management processes
are performed effectively and in accordance with controlling laws, rules, policies, and good
business practices.
1
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
Background
This audit was identified in the Office of Inspector General's (OIG) annual risk assessment and
included in the approved annual audit plan. It was performed in support of the Department's
goal of quality efficient services with the purpose of promoting the strategic imperative of
aligning financial resources with performance.
The purpose of the Division of Blind Services (DBS) is to work jointly with individuals, whose
primary disability is visual, toward achieving maximum levels of employment, independence,
and integration into the community.
During FY 2006-07, DBS served over 51,000 visually
impaired individuals. The program serves individuals either directly through its district field
offices or indirectly through contracts with community rehabilitation providers.
DBS programs and services are funded primarily from Federal grants. These funds, along with
State general revenue monies, totaled $48.6 million in FY 2006-07; nearly one-half (47 percent)
of this amount is spent on client services. In FY 2006-07, $11.8 million was spent to purchase
products and services for clients, while $11.1 million was paid for client services provided
through contracts with community rehabilitation providers. Products and services purchased for
clients includes, but is not limited to, computer equipment, software, low vision aids and
devices, physical and mental restoration, and training.
The majority of contracts are with community based rehabilitation providers for services that
include Independent Living, Vocational Rehabilitation (VR) Services, VR Transition Services,
Blind Babies, and Supported Employment Services.
Contracts provide for training and
counseling in various categories that include assistive technology, communication skills,
orientation and mobility, adjustment to blindness, and home management skills.
DBS has made advances in the administration of contracted and purchased client services with
the October 2006 introduction of AWARE (Accessible Web-based Activity and Reporting
Environment), a web-based case management system. The system covers the life cycle of a
DBS client from referral and application through eligibility, plan achievement, and case closure.
Designed to mirror the case management process, the system utilizes web technology to
automate essential functions and provides the ability to produce extensive reporting that can aid
in management decision-making.
2
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
Audit Objectives and Scope
This audit focused on evaluating DBS's procurement and management processes.
The
objectives were to ensure that: 1) client services contracts and purchase authorizations are
awarded and executed in accordance with controlling laws, rules, policies, and good business
practices; 2) clients receive services required by contract and purchase authorization terms;
3) payments are made in accordance with contract and purchase authorization terms; and
4) delivery of services and purchased items is properly administered and monitored. The scope
of the audit included contracts and purchase authorizations active between July 2006 and
December 2007, but included records outside that period that affected the review period's
transactions.
Methodology
To achieve the objectives, the audit team: 1) researched and reviewed applicable statutes,
rules, and procedures; 2) interviewed appropriate staff; and 3) reviewed selected contract and
authorization documentation active during the audit period.
Fourteen contracts and 69
purchase authorizations were reviewed based on a judgmental selection. Extended review of
support for additional authorizations was performed based on procurement and payment
processing issues noted during the field work phase.
Standards
This audit was conducted in accordance with the International Standards for the Professional
Practice of Internal Auditing, published by the Institute of Internal Auditors.
Audit Results
The findings below present the results of the audit as determined by: review of applicable
statutes, procedures and processes; interviews of auditee staff; and performance of tests and
other audit procedures. Each finding is organized into five segments, as explained below:
·
·
·
·
Condition The factual evidence found in the course of the examination.
Criteria The standards, measures, or expectations used in making an evaluation
and/or verification.
Cause The reason for the difference between the expected and actual conditions.
Effect The risk or exposure the Department may encounter because the condition is
not consistent with the criteria (the impact of the finding).
3
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
·
·
Recommendation Suggested course of action.
Management's Response Auditee opinion and statements regarding concurrence or
non-concurrence with the finding and corrective action planned by management.
Internal Controls
Improvements are needed in DBS contract and purchasing management. Internal controls in
some areas are weak or absent. This has allowed contracting and purchasing actions that may
have resulted in the Department purchasing unneeded equipment, and paying more than fair
market value for products and services.
deficiencies noted by the audit.
Contract Procurement
DBS contracts were procured on a non-competitive basis pursuant to specific exceptions
provided in statute.
During the 2006-07 contract period, 58 contracts were active with
Audit findings presented below discuss specific
approximately 20 different community rehabilitation providers. Two other contracts were issued
to provide services for clients at the Orientation and Adjustment Center in Daytona Beach. Our
audit noted areas where improvements can be made in DBS contract procurement processes.
Finding 1 Price analyses were not completed for contracts.
Condition: DBS has not performed and documented price analyses to ensure that prices paid
to contractors are fair and reasonable.
This is especially important for DBS
contracts because they are not competitively solicited, a process that helps ensure
price reasonableness.
We performed a unit price comparison of selected community rehabilitation
provider contracts by dividing the minimum number of clients to be served per
contract into the annual contract price. There are significant variances among
contractors in the computed cost per client served for three of the five contract
types - Independent Living, VR Services, and Supported Employment Services
contracts.
Contract cost per client ranged from: $891 to $1,626 (82 percent
variance) for Independent Living contracts; $1,974 to $3,739 (89 percent variance)
for VR Services contracts; and $4,500 to $30,938 (588 percent variance) for
Supported Employment Services contracts.
delivery of the same types of services.
The contracts compared involved
4
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
Variances in the unit price comparison could not be explained by DBS
management who commented that past contract prices were established based on
demographics of the service areas, specific circumstances of the individual
contract providers, and impacts of past negotiations with providers.
Criteria:
Section 216.3475, Florida Statutes (F.S.), requires that a person or entity awarded
funding on a non-competitive basis cannot be paid more than the competitive
market rate. Price and cost analyses are used to determine whether rates paid
are fair and at or below market value.
Section 287.057(5)(f)(7), F.S., provides an exemption to normal competitive
solicitation requirements, but requires agencies acquiring services to consider
reasonableness of prices charged by contractors.
Cause:
For two contract types, Blind Babies and VR Transition Services, a price analysis
by a former DBS employee was said to have been performed; however, there was
no documentation to evidence this. Analyses on other contract types also have
not been completed.
prioritized.
Effect:
Without price analyses, DBS has no assurance that prices paid to contractors are
fair and reasonable.
Recommendation: Price analyses should be prepared for all contracts procured on a noncompetitive basis.
reasonable.
contract files.
Management
Response:
Management is in agreement with the recommendation to develop a price
analysis prior to procuring all contracts. The Division of Blind Services will
begin such an analysis using a workgroup comprised of community
rehabilitation service providers and DBS personnel. The analyses will be
documented and maintained in the contract files.
Milestone: Begin
This will help ensure that prices are fair and
The need for price and cost analyses had not been
Such analyses should be documented and retained in
workgroup during July 2008 and complete analyses by September 15,
2008.
5
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
Finding 2 State purchasing laws were not followed.
Condition: A contract awarded during FY 2006-07 in the amount of $151,348 for nursing
support at a DBS residential facility expired on September 30, 2007, without
issuance of another contract. Services provided by the contractor were continued
after the September 30 expiration by using AWARE authorizations to pay vendor
invoices. This practice was continuing at the conclusion of our audit field work.
The Administrator of the residential facility was instructed to select clients in
residence whose individual plans for employment included physical and mental
restoration services.
The amounts of periodic vendor invoices were then
apportioned equally to each of these clients through AWARE authorizations.
Criteria:
When the purchase price of contractual services exceeds $25,000, purchases are
subject to competitive procurement practices per Section 287.057, F.S.
Procurement of such services should be evidenced by a written agreement
embodying all provisions and conditions of the procurement in accordance with
Section 287.058(1), F.S.
Cause:
Planning was not adequate to ensure a new contract was awarded before the
original contract expired. The decision was made to pay for contractor costs by
spreading costs over selected clients through the use of AWARE authorizations.
Effect:
State procurement laws have not been observed. No contract exists to control the
provision of services and the payment of fees. Use of AWARE authorizations for
payment of contractual services costs in this manner is improper.
Recommendation: A contract for the nursing services should be obtained as soon as
possible in accordance with the established procurement process. DBS
management
should
ensure
future
compliance
with
purchasing
procedures and laws.
Management
Response:
The intent of using AWARE authorizations was to capture costs at the
participant level for all service costs.
There is an exemption to
competitive procurement practices for health services in Section
287.057(5)(f)6, F.S. The DBS will comply with the recommendation and
is in the process of procuring a contract for nursing services effective
6
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
July 1, 2008. Management will ensure that future services are procured
with an executed contract versus using AWARE authorizations.
Contract Agreements
As a basis for their contract agreements, DBS utilized a standardized contracting format
established by the Department of Education (DOE) to ensure contract activities are executed in
accordance with Florida Statutes. Language specific to each DBS contract type was included to
adapt the agreements to their intended purpose. Certain revisions are needed to improve DBS
contracts.
Finding 3 Contract agreements need revisions.
Condition: We noted the following from our review of selected community rehabilitation
provider contracts:
·
Contracts do not reflect use of the AWARE case management system.
Introduction of this system changed important aspects of the contract
administration process that impacted client progress reporting, deliverable
reporting, and contract invoice processing.
·
Sanctions were not included in contract agreements.
Where practical,
contract agreements should contain monetary sanctions for nonperformance by the contractor.
·
Contract performance standards had not been incorporated into most
contracts. Only Blind Babies contracts included an attachment (Standards
and Indicators) that addresses the core contractor activities of intake,
evaluation, assessment, service delivery, reporting, and personnel
development.
Criteria:
Contract provisions and requirements should accurately reflect use of the AWARE
case management system including performance duties, deliverable reporting
methods, and contract invoice processing.
The Department of Financial Services' State Expenditures Guide, requires that
contractual services contract agreements contain sanctions for contract provider
7
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
non-performance.
Contracts should provide sufficient guidance to contract providers on what the
Division expects of them (i.e., performance standards).
Cause:
DBS staff did not update language in the standard contract templates with details
regarding use of the AWARE system.
Staff did not include provisions for
sanctions in the contract templates. Contract performance standards were not
completed in time to include them in contracts when they were originally issued.
Effect:
Use of the AWARE system changed aspects of contract administration, making
the unrevised contract language obsolete as it does not accurately reflect
reporting and invoicing activities required of contract providers.
For example,
deliverable reporting on client progress is no longer based on hardcopy reports,
but on electronic storage of case notes in the AWARE system.
The absence of penalties and sanctions in contracts reduces the Division's options
when performance is below the expected level or service quality is poor. Without
sanctions, there may be no mechanism in place to remedy the conditions noted.
Use of the performance standards can guide contract providers and serve as
criteria for Division staff in evaluating contract compliance and measuring
performance.
Contract monitoring procedures can include steps to verify that
services are provided in accordance with the standards.
Recommendation: a) Future contracts should require use of the AWARE case management
system.
b) DBS management should ensure that contract agreements include
specific
sanctions
for
non-performance
of
tasks
required
of
contractors.
Sanctions should include specific steps for prorating
contractor payments if minimum contract measures are not met.
c) Performance standards should be incorporated into all contract types
as soon as practical to provide guidance to providers and ensure
greater accountability over contractor performance.
Management
Response:
The Blind Babies contracts (these contracts are effective July 1 through
June 30) included language that mandated the use of AWARE for
8
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
entering client case notes and actual services provided to clients based
on their Individualized Plan. The service providers understood that actual
services and case notes were to be entered on all contracts. This contract
cycle, October 1, 2008 through September 30, 2009, all other DBS
contracts (Vocational Rehabilitation, Transition Services, Independent
Living and Supported Employment) will be revised to require use of the
AWARE Case Management System. Performance standards have been
developed for all contract types and will be incorporated into the contracts
in the next contracting cycle.
These standards were developed by
assigned workgroups composed of DBS and Service Provider employees
using an outside facilitator. Milestone: Include Standards and Indicators
Attachment in all contracts that do not have them (Vocational
Rehabilitation, VR-Transition Services and Independent Living) by
October 1, 2008.
Begin a workgroup for Supported Employment
contracts during July 2008 and complete a Standards and Indicators
Attachment by September 15, 2008.
There is no provision in the Florida Statutes to include contract language
that imposes remedies (sanctions or penalties) or rewards. The DBS has
cancelled several contracts over the last three years because of nonperformance of the contractor. However, the DBS will craft language that
identifies remedies, rewards and monitoring procedures and ensure the
DOE Contracting Office approves the new contract language prior to
including it in the current contracts. Milestone: Begin workgroup during
July 2008 and complete new contract language by December 31, 2008.
Finding 4 Contract requirements for client referrals need revision.
Condition: Community rehabilitation provider contracts (except for Supported Employment
Services contracts) include a restrictive clause regarding client referrals by DBS.
The contracts contain deliverable language that, in most instances, identifies the
minimum number of clients to be served. Contracts state that deliverables are
predicated on DBS referral of clients to the contract provider for various services.
Contract language further states that contract providers "will not be held
9
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
accountable" if the number of service referrals made by DBS does not meet or
exceed the number of eligible individuals indicated in the contract.
We determined that for most contract types, this condition is not realistic. For
example, during the 2006-07 contract term, DBS client referrals to providers of
Independent Living services averaged only 16 percent of total referrals; this is far
below the minimum number of client referrals stated in the Independent Living
contract.
Most referrals for Independent Living services came from eye care
We further
providers, family members, friends, and through self-referrals.
determined that DBS referrals to contract providers are not being tracked for
contract reporting purposes.
Criteria:
Contract agreements should contain reasonable terms and conditions to govern
the Division's relationship with contract providers.
Cause:
We were informed that the language was included in contracts when the Division
first began including minimum numbers of clients to be served as it was difficult to
predict the level of client referrals.
Effect:
Depending on the contract, the requirement represents an unreasonable
expectation on the Division's part. The language may weaken or prohibit DBS
from taking actions in response to underperforming contract providers (e.g.,
cancelling contracts, recovering contract funding, negotiating for lower contract
prices).
Recommendation: The referenced language should be revised or deleted from DBS
contracts where the provision is not reasonably attainable. Provisions
should be made to begin effectively tracking client referrals in the AWARE
system if DBS management determines this to be beneficial.
Management
Response:
The referral process will be further developed in concert with the
Community Rehabilitation Providers.
A likely solution will involve a
measure that reflects the Community Rehabilitation Providers requirement
to obtain referrals from their outreach activities and a DBS requirement to
have a referral measure. Milestone: Begin workgroup during July 2008
and complete new contract language by September 1, 2008.
10
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
Contract Performance
Evaluating contract performance, at a minimum, involves determining whether providers have
met or will meet the contract terms. In the absence of such an evaluation by DBS staff, we
compared contract performance deliverables for selected contracts to actual contract results
provided to us from DBS staff members.
Our review noted findings relating to contract
performance and the means used to report contract results.
Finding 5 Contract closeout was not performed.
Condition: A formal contract closeout process was not employed by DBS to evaluate the
performance of contractors and whether the Division obtained the services paid
for.
Criteria:
A process should be in place to establish that specific contract performance
requirements and deliverables have been met. Since most DBS contracts require
that a designated minimum number of clients be served, a comparison of actual
clients served to the contracted minimum should be performed and evaluated.
Cause:
Effect:
Closeout evaluation of contracts was given limited prioritization.
Contractor performance cannot be objectively evaluated in the absence of a
process established to determine whether contractors performed their duties and
provided deliverables per the contract terms.
If contractors are not held
accountable for their performance, there may be limited motivation for them to
provide the required or higher levels of services.
Recommendation: A documented closeout process should be routinely performed for all
contracts to determine whether the Division received services it paid for.
Results should be:
reported to executive management; used for
negotiations on future contracts; and, if applicable, used to assess
liquidated damages/sanctions for non-performance/non-compliance.
Management
Response:
The Division of Blind Services management will develop a checklist
document to assist in the closeout process of the contracting cycle and
include it in the DBS contract monitoring procedures manual. Milestone:
Develop a contract closeout checklist and revise the DBS contract
11
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
monitoring procedures manual to include a contract closeout checklist
during July 2008.
Finding 6 Many contracts resulted in fewer clients served than required.
Condition: We evaluated the actual results of selected 2006-07 community rehabilitation
provider contracts and found that most did not serve the minimum number of
clients required by the contracts. Based on our review of four to six contracts per
contract type and contract results provided by DBS staff, we determined that
contract providers in three of the five contract types under performed significantly:
Blind Babies Three of the four contracts served more clients than planned.
Overall (combined) performance was 20 percent above the minimum number of
clients to be served.
Independent Living contracts Each of the five contracts served fewer clients
than planned. Overall performance was 28 percent under the minimum number of
clients to be served.
Vocational Rehabilitation Services Four of the six contracts served fewer new
clients than planned. Overall performance was 34 percent under the minimum
number of new clients to be served.
Supported Employment Services Two of the five contracts resulted in fewer
job placements than planned.
Overall performance was 3 percent under the
minimum number of job placements planned.
VR Transition Services Each of the five contracts served fewer new students
than planned. Overall performance was 57 percent under the minimum number of
new students to be served.
Criteria:
Contract providers should serve at least the minimum number of clients projected
in the contracts. If fewer clients are served, DBS should determine and document
the cause(s) for the inadequate performance and take corrective action.
Cause:
Causes for contract under performance were not determined because a proper
closeout analysis was not performed (see Finding 5). Contract performance could
be impacted by inaccurate reporting of results. Certain results were self reported
12
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
by contract providers. Other results were based on data queries of the AWARE
case management system, performed by DBS technical staff.
Effect:
Though most contract providers under performed, fixed contract prices were paid
to providers as required in the contracts.
Assuming the results reported are
accurate, the Division is not receiving the services paid for.
Recommendation: DBS management should further analyze contractor performance for the
2006-07 contract period to determine why some contractor performance
was unsatisfactory.
Management should direct that more timely and
effective contract oversight be exercised to help ensure contract providers
serve clients in accordance with the contracts. Future contracts should
provide monetary sanctions for non-performance by contractors.
A
consideration for use of a contract type other than fixed price may be
prudent.
Management
Response:
The AWARE Case Management System was implemented on October 6,
2006. There were response time problems and issues involving the data
entry of Actual Services by service providers. This resulted in inaccurate
and incomplete reporting of results.
The design and response issues
were resolved during June 2008 when a new data entry module was
implemented. DBS management will ensure that an analysis of the 20062007 contract period is conducted to determine the trends in performance
for all contracts. New reports have been designed to identify contract
measures and results to assist in the contract analysis. Milestone: New
reports and analyses will be completed during July 2008.
Finding 7 Reporting of actual contract results is not accurate.
Condition: Contract results obtained based on a query of data recorded in the AWARE
system were reviewed and found to be inaccurate.
We compared AWARE
generated results for three of the five contract types to results that were self
reported by contract providers and found significant variances among the
contracts reviewed. On an aggregate basis, AWARE generated results for the
selected contracts fell under the self reported results in each case (28 percent
13
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
under for Blind Babies contracts, 23 percent under for Independent Living
contracts, and 19 percent under for Supported Employment Services contracts).
Criteria:
The AWARE case management system is established, in part, as a repository of
information on services provided to clients and progress observed. The system
should be adequately maintained to reflect this information and report aggregate
results in accordance with deliverables established in contract agreements.
Cause:
In reviewing the data with DBS staff, we noted instances where contract providers
failed to input actual services to the AWARE system, thus making reported results
inaccurate. Additionally, we found flaws in the reporting methodology used to
report actual contract results in the AWARE system. We were informed by DBS
staff that efforts to automate AWARE reporting of contract results were still in a
testing phase and that information reported was only as accurate as the
information input by contractor staff.
Effect:
Evaluation of contract performance is hindered without accurate information on
actual results achieved by contract providers.
DBS plans to eventually begin
using AWARE generated information for all contract performance reporting
(including federal reporting). This heightens the need for accurate reporting by the
system.
Recommendation: DBS management should provide guidance in the form of written
procedures and training to contract provider staff to ensure that contract
results are accurately input to the AWARE system.
validation of reported results may be needed.
DBS review and
Corrections should be
made to the AWARE reporting methodologies to ensure accurate
reporting of contract results.
Management
Response:
DBS will continue to provide training and follow-up technical assistance to
ensure that provider staff has the tools and knowledge to accurately input
data into the AWARE system. Procedures for entering Actual Services
were developed by the AWARE vendor, Alliance Enterprises, Inc., and
were provided during training prior to implementation of the AWARE
system. New reports have been designed to identify contract measures
and results to assist in contract analyses. Milestone: New reports will be
14
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
completed during July 2008 and service providers that require additional
training will be identified and scheduled for training during July 2008.
Contract Payment Processing
The AWARE system is used to process payments for community rehabilitation provider
contracts.
An electronic invoice is prepared based on services entered to the system by
Our review noted findings regarding support for contract payments,
contract providers.
expenditures to contractors for non-client costs, and overpayments for client services.
Finding 8 Contract payment processing can be improved.
Condition: Our review of 28 contract payments noted the following exceptions related to
invoice approvals, invoice documentation, and recording of client case notes:
·
·
·
Invoices for 7 of 28 (25 percent) payments were not approved by
designated contract managers.
Summary invoices for 8 of 21 (38 percent) AWARE generated payments
did not agree with the detailed support.
Invoices for 5 of 17 (29 percent) AWARE generated payments showed
prior month's service dates indicating that actual services were not being
input to AWARE on a timely basis by contract provider staff members.
·
Invoices for 4 of 21 (19 percent) AWARE generated payments did not have
invoice support as provider staff members failed to input any detail of
actual services provided.
·
In 5 of 9 invoices (56 percent) where AWARE was reviewed for the
preparation of client case notes, insufficient case note entries were found.
Criteria:
Contract managers designated in each contract should approve payments to
contract providers.
Invoice documentation should be sufficient to support all
payments. Community rehabilitation providers are required to input services they
provide to clients into the AWARE system in an adequate and timely manner.
DBS management informed us that contracts require case notes for clients
receiving services be input a minimum of one time per month.
Cause:
DBS staff at the Division Office approved invoices in place of designated contract
managers. Contract provider staff members did not input detail for actual services
15
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
provided to clients in an accurate and timely manner. Provider staff members also
did not sufficiently record AWARE case notes describing client progress as
required.
Effect:
Invoices not reviewed and approved by assigned contract managers may be
inaccurate.
Services provided to clients will be inaccurately reported unless
properly input to the AWARE system by contract provider staff members. DBS
staff cannot effectively monitor client progress unless case notes are input to the
AWARE system in an accurate and timely manner.
Recommendation: DBS management should ensure that contract payments are processed
properly. Efforts should be made to determine why AWARE generated
payments did not agree with the detailed support. Contract managers
should not approve contract provider invoices for payment unless they
have timely and complete support.
Operating procedures should be
prepared to direct both DBS and contract provider staff on how contract
payment processing should occur.
Additionally, written procedures
should be prepared to provide guidance in the preparation of AWARE
case notes.
Management
Response:
DBS management will ensure that contract payments are processed
properly. Additionally, DBS will provide guidance to assist DBS contract
manager staff as well as contract provider staff in processing payments
and invoices timely and accurately.
DBS will also provide written
guidance on preparing AWARE case notes and develop an invoice
activity report that will identify the number of days that have elapsed
between the DRAFT and SUBMIT cycle, between the SUBMIT and
APPROVE cycle and between the APPROVE and RELEASE for
PAYMENT cycle.
This report will assist all involved parties to track
invoice cycle time and adjust as needed. Milestone: Training, new reports
and written procedures will be completed by September 15, 2008.
16
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
Finding 9 Supplemental payments were made to contract providers.
Condition: AWARE authorizations were used to make supplemental payments to 12 contract
providers when a contract amendment should have been used.
Amounts
disbursed in these cases totaled $149,513 as of November 2007. Payments were
processed to look as though rehabilitation technology services were being
provided to clients, when they were actually payments to contracted providers.
Criteria:
Payments to providers should be made in accordance with contract payment
terms. Use of a contract amendment would be needed to alter provisions of the
contract.
Cause:
The payments at issue were processed to compensate providers for extra time
and costs they were said to have incurred based on problems encountered with
the AWARE case management system.
The decision was made to use
authorizations to facilitate the payments in place of amendments to the contracts.
Effect:
Payments were for costs directly related to the provision of contracted services,
but were not authorized by the contract agreement and caused the contract award
amount to be exceeded. The established contracting process was circumvented,
thereby bypassing the controls over the process. The method used to process
payments to contact providers was inappropriate as additional services were not
received by clients.
Recommendation: When necessary, contract amendments should be used to authorize
expenditure of contract funds in excess of the established contract
amount.
AWARE authorizations should only be used when payments
benefit specific clients.
Management
Response:
DBS has prepared contract amendments on current contracts to include
all contracted services. Milestone: Contract amendments will be effective
during June 2008.
Finding 10 Overpayments for client services occurred.
Condition: Training services provided to clients by a community rehabilitation provider were
billed based on an hourly fee rather than the established contract rate.
The
17
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
services, however, should have been fully provided for under an existing contract
established with another provider. As a result, the Division unnecessarily made
payments totaling $26,909 for services that should have been provided at no
additional cost by the provider under contract.
This condition was not identified from planned audit tests of purchase
authorizations, but was noted when performing audit procedures in other areas.
While this could have been an isolated occurrence, overpayments on other client
services may have occurred and gone undetected by our audit testing.
Criteria:
Client services contracts identify counties in which specific services are to be
provided. Contract providers should provide services in accordance with contract
requirements.
Cause:
The DBS rehabilitation supervisor who approved payments for services said she
was unaware that a contract was in place and should have been used.
Effect:
The payments charged to DBS on a fee for service basis represent overpayments
as the costs should have been included under the existing contract.
Recommendation: DBS should ensure that contractors provide services in accordance with
agreement terms.
Because this practice may be occurring in other
districts, management should communicate these requirements to all DBS
staff responsible for approving such payments.
Management
Response:
DBS will be reviewing the alignment of our current boundaries and
determine the most practical solution that will ensure that contractors are
adequately serving their designated districts.
Milestone: A review of
current contract boundaries will be completed during July 2008.
Contract Monitoring
Contract monitoring should be adequate to ensure required levels of services were received for
contract awards. As no formal monitoring was performed for DBS contracts, we focused our
emphasis on evaluating contract monitoring policies and procedures that were being
implemented. Our audit noted issues regarding the sufficiency of the monitoring process.
18
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
Finding 11 Contract monitoring needs improvement.
Condition: The Division has developed contract monitoring policies and procedures and
monitoring instruments to use in contract monitoring reviews, and District
Administrators (DA) have received training on contract monitoring activities.
Although DBS policy directs DAs to monitor contracts in accordance with
established procedures, DAs had not begun formal monitoring activities. We were
informed that they routinely performed informal monitoring tasks such as
maintaining regular contact and holding meetings with providers when necessary,
documenting communications relating to contracts, and approving monthly
invoices. They had not, however, performed on-site visits to contract provider
facilities
involving
use
of
established
monitoring
checklists,
interview
questionnaires, and other monitoring tools.
A two-person team in the Division Office's Bureau of Operations and Compliance
comprise a unit called the Compliance Review Section. This unit had recently
performed several compliance reviews of contract providers. Selected DAs were
participating in these reviews to obtain on-the-job instruction on how to perform
formal monitoring activities.
Criteria:
District administrators are designated as contract managers for community
rehabilitation provider contracts. DBS's policy entitled "Contract Monitoring and
Compliance Procedures" directs DAs to monitor provider contracts in accordance
with the DBS Contract Monitoring and Compliance Procedures Manual.
Section 287.057(15), F.S., makes contract managers responsible for enforcing
performance of the contract terms and conditions and ensuring that contractual
services have been rendered in accordance with the contract terms prior to
processing of invoices for payment.
Cause:
The Bureau Chief for Client Services and Support felt that DA relations with
contract providers could be negatively impacted if they are required to perform
duties as recorded in the procedures guide. He prefers that Division Office staff in
the Compliance Review Section perform formal monitoring.
Effect:
Timely and effective monitoring of contract performance cannot be attained
without the participation of DAs. The Compliance Review Section was essentially
19
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
established to oversee monitoring of contract providers by the DAs, as well as
conduct its own independent compliance reviews of contracted services.
The need for timely and effective monitoring of contracts is considered critical
given the nature of the contracts, the fact that services were being self-reported by
contract providers, and the basic deficiencies reported by recent DBS compliance
reviews (e.g., low service provision and closure rates, insufficient client progress
reporting, inadequate file documentation, and issues regarding staff certification).
Recommendation: DBS management should ensure that contracts are monitored in
accordance with established procedures.
Management
Response:
DBS management will ensure that contracts are monitored in accordance
with established procedures by the district administrators that are
designated as contract managers. Milestone: The district administrators
that are designated as contract managers will comply with the DBS
"Contract Monitoring and Compliance Procedures" effective immediately.
Operating Procedures
Effectively written operating procedures have many benefits that include clearly delineated
processes, greater consistency in operations, and establishment of internal controls over
business activities. Existing procedures are not adequately detailed to provide employees clear
and detailed guidance concerning core activities performed. This lack of procedural controls is
a cause for many of the findings in this report.
Finding 12 Procedures should be improved.
Condition: Procedure manuals for Vocational Rehabilitation Services and Blind Babies
primarily are comprised of policy statements rather than operational procedures
with sufficient detail to guide employees. Additionally, the Blind Babies manual
made reference to a previously used case management system.
Written
standardized operating procedures had not been prepared for core activities
performed by DBS employees. Examples include:
·
Provision of client products and services via the AWARE system (e.g.,
procedures on determining, authorizing, and documenting maintenance,
20
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
training, rehabilitation technology services, physical and mental restoration,
and other products and services).
·
Provisions for obtaining and documenting quotes for the purchase of client
products and services.
·
The administration and provision of services by contracted service
providers.
Criteria:
Written procedures are an essential component of the Department's internal
control structure that helps ensure management directives are carried out. They
should be prepared to address all essential functions and regularly updated.
Cause:
Procedure manuals had not been updated to reflect use of the AWARE case
management system, and had not been written with sufficient detail to serve as
operating procedures to guide and direct DBS staff members in the districts.
Effect:
Adequately detailed written procedures will: help ensure office activities are
performed in accordance with statutes, rules, and management's directives;
provide a basis for evaluating staff performance; and serve as a training tool for
new staff. Absent written operating procedures, management lacks assurance
that its directives will be performed.
Recommendation: Management should analyze all core activities performed by DBS staff
members in district offices and prepare detailed standardized operating
procedures that will guide employees on all activities important to the
Division's mission.
Management
Response:
DBS
management
will
work
to
develop
standardized
operating
procedures manuals to supplement the policies and procedures currently
developed and adopted.
Milestone: Standardized operating procedure
manuals will be prepared and promulgated by December 31, 2008.
Finding 13 Access rights to approve purchase authorizations need to be addressed.
Condition: AWARE purchase authorizations were approved by DBS employees who were not
authorized to do so. Additionally, individual approval limits for authorizations were
21
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
too high to provide reasonable management control of the process. Our review of
AWARE authorizations noted the following:
·
DBS employees in positions not authorized to approve authorizations were
doing so.
For example, 387 authorizations had been approved by
Other positions approving
administrative secretaries in two districts.
authorizations included rehabilitation technicians, rehabilitation specialists,
and a word processing systems operator.
·
Division Office staff members providing technical support for the AWARE
system were given authorization to approve authorizations.
·
Approval limits were set at $50,000 per authorization for district employees
and $1,000,000 per authorization for selected employees at the Division
Office, including the technical support positions noted above.
Criteria:
AWARE authorizations should be approved by employees in specified positions,
principally the District Administrator and Rehabilitation Supervisor positions. Other
employees at the Division Office are authorized to approve authorizations in a
backup capacity.
The AWARE system is programmed to allow approval of
individual authorizations up to a limit designated in the system. We noted that the
average authorization amount approved was under $500 and only one
authorization exceeded $50,000 during the 2006-07 fiscal year. Limiting approval
authority for expenditures is an important control that strengthens management's
oversight of purchases.
Cause:
DBS staff indicated that individuals who should not have had system rights to
approve authorizations had apparently been "overlooked" when system security
was tightened in January 2007. Several staff members at the Division Office,
including three technical support staff, were given system rights to approve
authorizations in a backup capacity.
streamline the authorization process.
Effect:
Management controls are weakened or eliminated when approval authority is set
too high or not limited to the appropriate employees. This can result in increased
risk that inappropriate transactions occur. Technical support staff members who
The approval limits were set high to
22
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
have programming access to the AWARE system architecture should not have the
capability to approve authorizations.
Recommendation: a) DBS management should adopt written procedures to ensure only
authorized staff are given AWARE system access to approve
authorizations.
b) Approval access should be limited to employees who have a working
knowledge of the specific transactions being approved. Approval of
authorizations by the Division Office in a backup capacity should be
limited. AWARE technical support employees should not be given the
ability to approve authorizations.
c) Transaction approval limits should be re-evaluated and set to lower
levels.
Management
Response:
DBS management re-evaluated transaction approval limits and lower
levels have been established for all approvers. Approval rights have been
delineated to designated personnel in the AWARE system. Also
management has identified a primary person to serve as backup in the
event of a necessary emergency approval. Milestone: Completed during
May 2008.
Non-Client Purchases Using AWARE Authorizations
The AWARE authorization system was designed to provide products and services for individual
clients either through direct payments to the clients or through payments to vendors who
provide products and services to clients.
However, AWARE also was used to facilitate
payments for products and services not directly identifiable to specific clients. The following
finding includes examples of such payments and demonstrates the lack of operating procedures
to guide and direct DBS staff on the types of purchases to be made using the AWARE system.
Finding 14 Purchasing practices need to be improved.
Condition: AWARE authorizations were used as an expedient way to purchase products and
services, some of which should have been procured using direct orders or formal
contracts. We noted the following questionable uses of AWARE authorizations:
23
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
·
Fixed monthly payments (totaling $90,667 from December 2006 through
December 2007) were made to a community rehabilitation provider for
follow up services to clients whose DBS cases had been closed.
Payments had previously been funded using the DBS Grants and
Donations fund. Procurement of services was not awarded using a formal
contract.
·
Fixed monthly payments (totaling $80,000 from January through August
2007) were made to two vendors for delivery of peer support services.
Cooperative agreement formats were used in place of contracts.
The
cooperative agreements did not incorporate the provisions of formal
contracts (e.g., designation of a contract manager, sufficient descriptions
of performance duties, deliverable reporting, payment terms, requirements
for staff qualifications, incorporation of standard terms and conditions, etc.)
One of the vendors received a $20,000 payment described as "seed"
money, which was not in accordance with the cooperative agreement
provisions.
·
One vendor was paid for presenters who provided DBS staff training at
several events from December 2006 through June 2007. None of the
seven individual payments was greater than $25,000, the threshold for
competitive bidding requirements, but the seven payments totaled
$95,069, well above the threshold for competitive bidding.
quotations were not obtained for these services.
·
Advance payments of $70,000 were made to a state university for hosting
a sports camp for Vocational Rehabilitation clients. DBS staff reportedly
chose the provider after reviewing several other potential providers,
however, no written documentation of this or of any quotes obtained was
prepared. A non-standard contract was used, which was signed by a DBS
program consultant who did not have signatory authority. Services by the
university were begun before the contract was executed.
·
Equipment described as demonstration units for community rehabilitation
centers around the state was purchased using authorizations. The cost of
most of the individual equipment items was greater than $1,000, but these
24
Price
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
items were not accounted for as tangible personal property (i.e., tagged,
assigned to a property custodian, and tracked as a part of the official
property inventory).
·
AWARE authorizations were issued after receipt of product or service
invoices, and not as pre-authorizations of purchases.
Criteria:
·
When the purchase price of commodities or contractual services exceeds
$25,000, purchases are subject to competitive procurement practices.
·
Purchases considered exceptional (e.g., single source) or exempt (e.g.,
services to persons with disabilities) must have those conditions
documented and be made in accordance with provisions of Section
287.057, F.S. Procurement of contractual services greater than $25,000
is to be evidenced by a written agreement embodying all provisions and
conditions of the procurement of such services (Section 287.058(1), F.S.).
·
Use of cooperative agreements for the procurement of contractual
services is not addressed in DBS policies.
·
Florida Statutes prohibit advance payments to vendors except in certain
circumstances, none of which apply in the instances noted.
·
Section 287.057(10), F.S., provides that an agency shall not divide the
procurement of commodities or contractual services so as to avoid
requirements for competitive sealed bidding.
·
Contracts that bind the Division can only be signed by the Commissioner
of Education.
·
Section 287.058(2), F.S., provides that contractual agreements over
$25,000 in value must be executed before services are rendered.
·
The Division is required to identify, manage, and control state-owned
tangible personal property in accordance with applicable laws (Section
273.02, F.S.), and rules (Chapter 69I-72, Florida Administrative Code).
Cause:
AWARE authorizations were used to expedite payments for various products and
services. DBS policies and procedures are not in place to provide guidance to
25
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
staff on the types of purchases that are allowed using the AWARE system.
DBS does not have a policy to direct staff on the use of cooperative agreements.
They appear to have been used in the instances noted to facilitate payments to
vendors in place of issuing formal, more restrictive contracts.
Regarding the expenditures for staff training, we were informed that a former
division director instructed that specific persons be used as training consultants.
No attempts were made to determine whether other vendors were available or
whether fair prices were being charged.
The non-standard contract for hosting of the sports camp for Vocational
Rehabilitation clients was signed by a DBS program consultant who was not
aware that he was not authorized to do so. Activities related to the camp were
begun before a contract was prepared and signed.
DBS staff members were unaware that equipment purchased for demonstration
(not for individual client use) should have been accounted for as tangible personal
property.
Effect:
Contractual services may not be acquired based on a system of full and fair open
competition, which helps ensure the Division pays a fair and reasonable price for
services obtained. The appearance and opportunity for favoritism is increased,
and public confidence that contracts are awarded equitably and economically is
compromised.
Without a contract containing appropriate provisions and conditions as required by
law, the Division may not be able to maintain adequate control over the services
provided, the payment process, the monitoring process, etc.
Unauthorized advances to providers may result in the Division paying for services
not received.
The contract for hosting of the sports camp for Vocational Rehabilitation clients
included an assertion that the person who signed on behalf of DBS had full
signatory authority; the employee, however, did not have this authority. Allowing a
contractor to provide services before a written agreement is executed creates risks
to the Division.
26
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
Control of tangible personal property is weakened when it is not recorded in the
State property system. Equipment can be lost or stolen without detection.
Use of AWARE authorizations as noted above circumvents the usual procurement
methods and the controls designed in these methods.
Recommendation:
a) AWARE authorizations should only be used for case management
when there is a specific individual who will be served or benefited.
The DOE Purchasing Administrator should be contacted for
advice on procurement of client products and services.
b) Policies and procedures should be prepared to guide DBS staff on
the types of purchases to be made using the AWARE system. To
maintain
an
effective
level
of
internal
control,
AWARE
authorizations should be used only in a case management
capacity when there is a specific client that will be served or
benefited.
c) DOE guidelines should be followed regarding appropriate
procurement methods to use. Direct orders or contracts should be
used when appropriate.
d) DBS should develop policy regarding use of cooperative
agreements in procuring client services.
e) Advance payments to vendors should not be made unless
authorized and in accordance with Florida Statutes.
f)
Contracts should only be signed by the Commissioner of
Education or a person who has been formally delegated to sign for
the Commissioner.
g) State-owned tangible personal property should be accounted for
in accordance with applicable laws and rules.
Management
Response:
DBS management concurs with all recommendations for improving
purchasing practices. DBS is in the process of preparing policies and
procedures that will address the following: AWARE authorizations
utilization, appropriate purchases using the AWARE system, how and
27
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
when to use cooperative agreements, and authorizations for advance
payments. Milestone: DBS policies and procedures on purchasing will be
prepared and promulgated by June 2008.
Purchase Authorization Payments
The AWARE authorization process provides the means for purchasing and requesting
payments for client products and services. The system was designed to process purchases for
individual clients in a case management setting. In a review of selected payments, we noted
findings regarding the timing and support for authorizations issued. We noted two other findings
related to payments for client services.
Finding 15 Payment processing for purchase authorizations should be improved.
Condition: We reviewed 69 selected purchases made using AWARE authorizations. These
payments were for such products and services as equipment, cash assistance
(maintenance), school tuition, low vision aids, and medical procedures. We noted
the following irregularities:
·
Authorizations for 23 purchases were issued after products or services had
been received.
·
Nine of 10 purchases involving payment of maintenance assistance to
clients did not have adequate support for amounts paid or sufficiently
documented justification by DBS staff.
Criteria:
Authorizations are essentially purchase orders issued for the acquisition of
products or services and if accepted, serve as a legal commitment to which both
parties must adhere. As such, these instruments should be issued and accepted
before they are authorized to provide products or services.
Maintenance payments to clients should have adequate support for amounts paid.
Justification for payments should be sufficiently documented.
Cause:
Written operating procedures have not been prepared to ensure activities are
performed in accordance with statutes, rules, and management's directives. DBS
staff did not always issue authorizations in advance of products or services.
28
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
Support documents or AWARE case notes were not always prepared to support
and justify amounts paid to clients as direct maintenance.
Effect:
Authorizations issued before products or services are provided bind the vendor or
contractor to the terms established and approved by DBS staff, and circumvent
the controls inherent in the approval process. Payments to clients that are not
adequately supported and justified may represent improper payments, payments
for an incorrect amount, or payments for an unauthorized purpose.
Recommendation: DBS management should ensure (preferably via written operating
procedures) that purchasing tasks are performed properly. Authorizations
should be issued before products or services are initiated. All purchases
should be documented with adequate support and justification for
amounts paid.
Management
Response:
DBS management will develop standardized operating procedures that
will address the recommendations listed. Milestone: Standardized
operating procedure manuals will be prepared and submitted to the OIG
for review by June 30, 2008.
Finding 16 Payments of cash advances to clients by contract providers occurred.
Condition: Contract providers made cash advances (maintenance) directly to clients, and
then billed DBS for reimbursement. We identified approximately $56,000 in such
payments for the twelve months ended September 30, 2007, to reimburse three
contract providers. Additional amounts may have been paid by other contract
providers. We were informed by DBS management that this practice has been
halted.
Criteria:
The established process for issuing maintenance payments involves DBS
disbursements based on an approved authorization which results in the issuance
of a state warrant payable to the client. The client is to sign a receipt evidencing
acceptance of the warrant when received. Allowing contract providers to make
such payments circumvents the controls inherent in the established process.
Cause:
Written operating procedures are not in place that direct how such payments are
to be performed. District administrators contacted explained that this practice was
29
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
being allowed as state warrants took longer to process.
Effect:
This practice has inherent risks involving the handling and transfer of cash that
could result in misappropriation of funds. Maintenance request forms were not
being prepared by clients as they would have been if processed by DBS staff.
Recommendation: Written procedures should be prepared to direct DBS staff on how
maintenance is to be processed.
Management
Response:
DBS will develop standardized operational procedures to guide staff on
how maintenance is to be processed. Milestone: Standardized operating
procedure manuals will be prepared and submitted to the OIG for review
by June 30, 2008.
Finding 17 Equipment purchases were made in advance of needs.
Condition: Computer equipment purchases were made without an identified need.
One
district purchased four laptop computers using names of clients that were never
intended to receive the equipment. This was done in order to have a backup
supply of computers.
The equipment was eventually distributed, though six
months passed before one laptop was documented to have been assigned to a
client.
Criteria:
The Vocational Rehabilitation Program Procedures Manual states that tools,
equipment, initial stocks, and supplies may be purchased for an eligible individual
when they are necessary to achieve an employment outcome.
The Bureau Chief for Client Services and Support indicated that such items are
not to be purchased in advance for a future use, but for a particular client at the
time of need.
Cause:
The district administrator responsible for purchase of the equipment did so based
on issues regarding compatibility of adaptive software with a newly released
computer operating system.
Effect:
When equipment is purchased using a different client's name, client files do not
accurately reflect products and services provided to them. Using client's names
as such equates to falsifying Division records. Stockpiling of equipment for future
30
Audit of DBS Contracted and Purchased Client Services
Audit No. 07/08-01A
use puts the equipment at risk for being lost or stolen, and spends State funds
before necessary.
Recommendation: DBS management should adopt policies and procedures that ensure
purchases of equipment are made only for eligible clients whose case
files support the need for such equipment.
Management
Response:
DBS management will adopt policies and procedures that ensure
purchase of equipment are made for eligible cases only. Milestone:
Standardized operating procedure manuals will be prepared and
submitted to the OIG for review by June 30, 2008.
Closing Comments
The Office of Inspector General would like to recognize and acknowledge Division staff for their
assistance during the course of this audit. The assistance provided by staff of both the DBS
and DOE is appreciated. Audit field work was facilitated by the cooperation of all personnel
involved.
31
Department of Education
Office of Inspector General
DBS Management Review
Issues resulting from interviews (primarily with former DBS employees)
1) Allowing contract service provider to use DBS facilities in Daytona Beach
at no cost - A client service provider, Center for the Visually Impaired, occupies
two buildings on the rehabilitation center campus, Buildings 1185 and 1187
located on Dunn Avenue. Campbell said the provider is essentially occupying
one or both buildings "for free" meaning they are not paying rent or utilities.
2) Expensive equipment abandoned at Rehabilitation Center in Daytona
Beach Former DBS employee saw much waste associated with the
construction of the new facilities at the center. An example of waste that cited
was the purchase of a $28,000 generator that is sitting at the center and is not
being used. Employee indicated he received instructions from the DBS
Executive Director to leave the equipment alone (hidden) for 5 years upon which
time it can be surplused (written off).
3) $30,000 "loan" from DBS Foundation - Per the Chairman for the DBS
Foundation, when Joyce Hildreth became the Exec. Director of DBS in 2009, she
met with the DBS Foundation requesting $30,000 which was needed to "make
payroll". Accounting of the $30,000 from the foundation was requested from
Joyce since then, but she has never given a response. The check was number
1026 and dated 3/5/2009.
4) Transfer of Legislative funding for Blind Babies to pay for two new
contracts that will serve adult clients - Approximately $29,601 of general
revenue funds slated for the blind babies program is going to fund two new
contract providers who have requested contracts with DBS for serving older blind
clients. The DBS received funds in new general revenue funding from the
legislature ($540,891) specifically for blind babies and it is inappropriate for this
to occur. The funding transferred to the new contract providers was taken from
case service dollars reserved to purchase medical procedures and supplies for
blind babies.
5) Gifts and Donations policy have been altered from prior OIG audit - The
prior audit called for the process of determining what was funded out of the gifts
and donations fund to be independent. The DBS Foundation was to be an
integral part of the grant process. This has been "undone" by the DBS executive
director as she has taken it internally with DBS employees and herself making
the determinations.
6) DBS contract planning/ development meeting attended by CRP staff
members - During recent contract development negotiations, leaders from
contract providers and a representative of the Florida Association of Agencies
Serving the Blind (a lobby group for contract providers) openly participated in the
process. Employee felt that it was odd to allow such representatives to attend.
She said this is an example of how certain contract providers control aspects of
the Division.
7) Repaid CRP for collected contract penalties (from G&D trust fund)
Remedies were applied to a 2010-11 VR contract. The provider objected to
paying the remedies. The remedies were collected, however, the executive
director reimbursed the provider with a check written on the DBS Gifts and
Donations fund for $22,186. Such a disbursement appears a questionable use of
the funds.
8) DBS Credentials Committee - DBS committee that hears requests for waivers
of certification requirements for people providing direct services to clients.
Having such a committee (especially one that is staffed with 2 CRP leaders) is in
essence allowing the CRPs to hire anyone they want. He said that this is how
DCF got in so much trouble.
9) Land swaps - Former DBS Maintenance Superintendent, stated that the DBS
Director should not be directly involved in "land deals" with rehabilitation center
property. He explained that she was involved with trading a large area of the
Daytona Beach Rehabilitation Center land for a much smaller portion of Daytona
State College land. He said that the State College needed space for a parking
garage.
Audit No.: M-11/12-20
Prepared By: ___KK _____
Date: ____5/3/12_____
Review Phase: Fieldwork
Purpose: Summarize review of Personnel Actions
Personnel Actions data dump
Reviewed By: __________
Date: __________
Approximately 300 DBS employees, with almost 800 personnel actions from 1/1/2009 to 4/7/12
· 2009 181
· 2010 250
· 2011 303
· 2012 - 50
Condensed results:
Transaction Description
Appointment
Demotion Appt
Promotion Appt
Reassignment Appt
Employee Sync
Pay Change
Involuntary Separation
Voluntary Separation
Grand Total
2009
43
2
1
17
0
19
2
27
111
2010
80
1
6
31
19
8
6
56
207
2011
90
3
11
29
17
24
10
62
246
Notable trends:
· Personnel actions have increased over the years.
· Voluntary Separations have increased from 27 in 2009; 56 in 2010; and 62 in 2011.
· Involuntary Separations have increased from 2 in 2009; 6 in 2010; and 10 in 2011.
Management Review of the Division of Blind Services
I:\AUDIT PROJECTS\2011-2012 Projects\DBS Management Review\2. Fieldwork\Personnel Actions\DBS MR write
up_PARs_I9.1.doc
Audit No.: M-11/12-20
Page 1 of 2
Number of positions advertised in People's First:
All (7/26/10 to 4/5/12)
Count of Requisition ID
Status
Total
Canceled
31
Filled
51
Open
11
Grand Total
93
For 2011 (only full year of data):
Count of Requisition ID
Status
Total
Canceled
18
Filled
34
Open
4
Grand Total
56
·
·
·
90 appointments (from data on the first page)
56 positions advertised
62% advertised
File Review:
·
Appointments Sampled 5 appointments 3 judgmental and 2 random.
o People's First advertisement
3 of the 5 were not advertised in People's First
2 of the 5 were advertised in People's First; however, 1 of those postings
was cancelled and the position was not filled from the People's First
posting.
o Internal advertisement
3 of the 5 were advertised internally by email
o Selection documentation
No documentation available to support why the candidate selected was
the best for the position.
Separations - Sampled 5 separations 3 judgmental and 2 random.
o Approval documentation
Involuntary separations had Notice of Final Agency Action letter signed
by the Deputy Commissioner.
Separations appear to have followed the appropriate levels of approval
(General Counsel, Labor Relations, and Deputy Commissioner).
3 of the 5 did not have clear evidence that the employee was aware of
job performance concerns.
·
Management Review of the Division of Blind Services
I:\AUDIT PROJECTS\2011-2012 Projects\DBS Management Review\2. Fieldwork\Personnel Actions\DBS MR write
up_PARs_I9.1.doc
Audit No.: M-11/12-20
Page 2 of 2
ûï¿ûðh
More information about the NFBF-L
mailing list