2001 Conference Proceedings, June 11-14, 2001

Assessing Student Fluency in Information Technology

Peter Smith
Saint Mary's College
Notre Dame, IN 46556
219-284-4493
psmith@saintmarys.edu

Abstract

In a society in which almost every college graduate needs to be fluent in the use of technology to gather and evaluate information, it is irresponsible for colleges not to ensure that students can meet these information technology challenges. The National Research Council has defined IT fluency to include the understanding of IT concepts and ability to handle high level thinking as well as possession of IT skills. This paper describes the process used by Saint Mary's College to help students assess their fluency level and set goals to ensure they are fluent by the time they graduate.

Introduction

This student fluency project arose out of the brainstorming a group of us at the college were doing to start a Teaching, Learning, and Technology Roundtable (TLTR). We defined the mission of our TLTR and agreed that the most pressing issue we were facing was student, faculty, and staff fluency in information technology as described by the National Research Council [1] and the Association of College and Research Libraries (ACRL) [2]. Since the faculty and staff fluency problem was quite daunting, we decided to focus on student fluency first. The entire effort is documented on the web at www.saintmarys.edu/~psmith/fluency.html [3].

A group of faculty and staff from the TLTR steering committee, student affairs, and the faculty department liaisons met in September, 2000, to discuss the issue of student fluency with information technology, how to assess it, and how to make sure that every student who graduates from Saint Mary's can function effectively in a technologically-oriented world. Some of the issues we considered are as follows:

  1. Students come to us at a variety of levels of IT fluency. We can say with almost certainty that no arriving student is any longer computer phobic. (The need to communicate via email and use a word processor has forced everyone to overcome their fear of computers).
  2. Although students are comfortable using the internet, they all need to learn how to separate the wheat from the chaff. This analysis of web information must be part of IT fluency
  3. Students appear all along the continuum from novice to experts in understanding and using information technology. To assess where they are, we need a rubric which identifies different levels of fluency and some identifying characteristics of each level.
  4. Many (perhaps most) students are motivated to improve their level of fluency on their own once they are aware of the possibilities.
  5. It is important to create an atmosphere of high expectation and support rather than a focus on requirements.
  6. In addition to classwork opportunities to improve fluency in information technology, we must create co-curricular opportunities, such as an SMC TV station, a video yearbook, or student grant projects. Perhaps we can create support groups (like RESNET) for various needs in the Residence Halls so students can improve their fluency by helping others.
  7. Bringing back graduates, or using them in other ways to exhibit the level of fluency needed in the workplace, would motivate students.
  8. We may be legally obligated to make sure students use the internet wisely and understand the consequences of their actions on line.



We brainstormed a number of ways to ensure that each student improves her level of information technology, but did not attempt to rank these or even to discuss them in depth. Some of the ideas are as follows:

  1. Have each student create a technology portfolio into which she puts evidence to support her assessment of her fluency level.
  2. Have each student develop a project (such as a sophisticated web page), with her advisor, which she works on during her four years and which constantly challenges her to improve her IT fluency.
  3. Give students a technical exam administered by a national testing service.
  4. Require a course or sequence of courses through which students learn IT concepts and improve IT skills.

One constraint that kept coming up was time. Whatever we decide to do will come on top of an already very intense course load for both students and the faculty who must assess their fluency. For example, the curriculum committee will not allow any additional course requirements without removing one of the current requirements. Since the strategic plan curriculum task force is currently studying the curriculum, it is a good time to be making IT fluency proposals.

Creating a Rubric for Self Assessment (Appendix 1)

We decided to spend a month trying to set up a rubric to be used in assessing a person's level of IT fluency. We compared the rubric to a ladder which specifies the rungs that a student can climb to reach higher levels of IT fluency. We asked if the rubric was a single ladder for all students or was really many ladders, one for each discipline. We agreed to push for the single ladder approach since the multiple ladder theory was so complex that it seemed impossible to define. Concern was raised that a work load similar to that to handle our two-tiered writing requirement will be placed on faculty if we are not careful in implementing an IT fluency assessment process. It may be a violation of academic freedom to push faculty to incorporate more exposure to deeper levels of technology in their courses. We did agree that all students should leave the college with a certain level of expertise in using technology so they can function as confident knowledge workers in today's high-tech society. The problem is how to accomplish this goal.

We asked the question, "what should the rubric be used for?" Should it represent guidelines to help students, their advisors (both academic and non-academic), the Dean, the curriculum committee, the mission committee, the strategic plan committees, etc., make plans to help each student reach her IT fluency goals? We agreed that the rubric should serve all the above constituencies. Technology needs to be consciously considered in advising students, in constructing the curriculum, in co-curricular activities, everywhere on campus.

After three major and many minor revisions, we developed the rubric found in Appendix 1. After it was approved, we discovered that a number of items in the rubric (e.g. email, which is listed in the beginning user skill column) have a variety of skill layers, some of which should be listed in the Passive, Active, and even Expert User rows. In some sense the rubric is really three dimensional and we decided to make these multilayered terms hyperlinks to background documents explaining how they apply at each of the user levels. I will demonstrate this on-line in my presentation. [3]

Difficulties with Assessing Fluency

We asked how we should push IT fluency. A single course for all students, perhaps an extension of our four week technology orientation experience that first-year students take now was rejected as not providing continuous "just-in-time" learning and as being very hard to set up and coordinate. Diffusing IT fluency experiences throughout the curriculum, much like writing across the curriculum, is very hard to assess because no one is accountable for making sure the experiences are of high enough quality and that each student avails herself of them sufficiently to achieve a high enough level of fluency. The best home for the assessment of IT fluency seemed to us to lie with the advisors. We asked how the rubric might be used by advisors -- i.e., as a talking point, or as a guideline, or perhaps a promise that if a student reached the fluency level she and her advisor agreed was optimal for her then she would be successful after graduation. It was mentioned that the local technical college actually backs up this promise with free tuition for courses taken after graduation if a student does not succeed in finding a suitable job.

If advisors are to take on the responsibilities mentioned above, they have to know where in the curriculum and in the cocurriculum to find opportunities for their advisees to grow their IT fluency, and there has to be a strong support mechanism in place to help them give good advice. If the rubric is to be really helpful to advisors and students as well as the other constituencies mentioned above, it needs to be simple to use. To accomplish this, we decided to put the rubric in grid form. This will emphasize the fact that no one finds herself totally at a single level in the rubric. At different times and in different settings she can be at one level in information literacy, another in skills, etc.

Student Information Technology Fluency Stages

Our overall goal is to make sure that students are prepared to enter the technologically changing world when they leave the college. They should have become lifelong learners who remain fluent with technology throughout their lives. We believe that this goal can be accomplished through the following four stages::

  1. Empower each student to self assess her fluency level in dialog with her advisor and to set goals for herself establishing the level she wants to attain by the time she graduates;
  2. Provide the student and her advisor with the knowledge to lay out a plan to reach her goals by inventorying the curricular and cocurricular opportunities available at the college to help her move up the fluency ladder;
  3. Provide ways that a student can demonstrate her competence with IT fluency;
  4. Provide a recognition system for those who have reached their fluency goals.



There was some disagreement as to whether or not we would have to implement all four stages before getting started. Some saw the learning that would happen as we implemented each stage would help define the next stage's implementation. Others felt that students would not take the program seriously unless we implemented all of the stages. We decided to implement the first two stages for next year (2001-2002) as a pilot project with students from the departments whose faculty become early advisors of students (e.g., Biology, Chemistry, Math, Nursing, perhaps Education and Business) and also some students who would normally be advised by the Office of First Year Studies or the Assistant to the Dean until late in their Sophomore year. We will try to convince advisors of this pilot group to use the rubric as a self assessment tool as described above. Using this pilot experience as a guide, we will then decide whether or not to implement all four stages at once.

It seemed clear that we would have to have in place by fall 2001 at least a beginning inventory of ways students can improve their fluency skills. It was also clear that we will need a support system for advisors as they started working with the rubric with students.

Resource List for Improving Fluency

The second stage in our plan to set up a program for improving fluency was to prepare an IT resource list of courses and extra-curricular opportunities that students can use to improve their IT fluency. Each department liaison was asked to prepare such a list. Student affairs and the residence hall staff helped with the list of co-curricular opportunities.

We brainstormed general categories of resources we will need to gather to help students improve their fluency level. These were:

  1. Courses - especially those that require use of a certain technology
  2. Workshops - a listing of their content will be necessary
  3. Student government positions and committees that use technology
  4. Summer jobs involving technology
  5. Internships involving technology
  6. Student employment involving technology
  7. Clubs that produce newsletters and/or maintain web pages
  8. Local on-line resources re technology
  9. Courses or workshops at other area colleges in our consortia
  10. Web on-line universities or tutorials
  11. Tutorial software on CDs (e.g. the one for Office 2000)

Survey Form for Faculty

We prepared a survey instrument (Appendix 2) for faculty to complete, listing their lower division courses for the 2001-2002 academic year and how they incorporate different kinds of technology in course assignments. The team working on the co-curricular opportunities prepared a list with a partial set of computer skills which were utilized by students in various student government and student employment positions. (Appendix 3)

As we were discussing the resource list we came to several realizations which are listed as follows:

  1. Students will teach each other. It is not necessary for either faculty teaching a course requiring student use of technology or supervisors of student government or employment positions to teach students about technology, but these faculty and supervisors should know where students can go to learn the technology.
  2. We are trying to create a culture at Saint Mary's in which students will take responsibility for their own learning, but in the meantime we need a process that will not just expose students to technology, but will insist that they make use of it.
  3. A survey form might uncover resources which help improve the technology skill level, but may not get at the information literacy component in the rubric.
  4. It is very hard to teach students to evaluate web-based information resources because they do not have the background in the discipline to call upon. It is hard enough for experienced faculty to evaluate these resources. We spent a good portion of time discussing what is reasonable to expect from students in their attempts to evaluate the quality of web-based information without reaching any consensus.

We concluded that the survey instrument we use to gather the resource list should focus on what students are expected to do in the course concerning the growth of their IT fluency, not on the technology the faculty member uses to transmit the course information. For each course or other opportunity, three questions need to be asked:

  • What kinds and levels of technology were students required to use in the course or position?
  • What specific assignments were they given requiring use of technology?
  • What were the learning goals of these assignments?

There was a healthy skepticism expressed that the survey would not uncover resources to help students improve their higher level critical thinking and assessment skills. Because we had a difficult time articulating exactly what we wanted from faculty who were to fill out the survey, we decided that we had better present it to the department liaisons in person at a meeting soon after spring break.

While we were discussing the survey form, a Biology teacher had filled in the grid for one of his courses to illustrate that it might not be as hard for faculty to work with as we might think. In fact, the resources he identified would be very apropos for any liberal arts course. We suggested that each of us take one of our courses and do the same so we would have a set of examples when we talked to the liaisons and they could use them when talking to the faculty in their departments.

The design of the survey form was hindered by the failure to specify the organization of the resource list that we will prepare for students and advisors. We decided to prepare an indexed list of identifiers and descriptions that faculty could use to categorize their course assignments regarding which aspects of IT fluency they enhance. The George Mason report [4] was suggested as a model for identifying categories. This report can be found at www.educause.edu/ir/library/pdf/eqm0041.pdf (specific goals on page 7). We revised this document for our purposes and called it a Glossary of IT Identifiers. (Appendix 4).

Each item in the glossary has an single-letter IT Identifier code which faculty members are to put on the survey form next to the course which incorporates this type of technology in its student assignments. We included a code 'X' which indicates that none of the IT Identifiers apply and the faculty member needs to use the Comment column to explain how the assignment enhances IT fluency. Note that there is a column on the survey form for the faculty member to identify representative assignments using the indicated category of technology. We hope that this column will help us identify the level(s) in the rubric on which we should list the course.

We discussed making the resource list available as links from the rubric, where a course would appear in the block(s) where it helps a student progress down the ladder. (I.e., it would not be listed at a level beyond which its opportunities would take the student, nor would it be listed in earlier levels where it would be too hard for students to master.) This suggestion was met with skepticism from some participants who believed that most resources would be multilayered.

Challenges Encountered in Trying to Set Up the Program

At the liaison meeting we encountered some of the challenges mentioned below, but found that most departments were willing to work with the survey form. At this meeting, we decided to restrict the survey to 100-200 level courses, since these are open to all students. Even if a student found a major course in another discipline which seemed likely to help her improve her IT fluency, it is unlikely that she would have the prerequisites to take it.

A number of faculty felt that they need to improve their IT fluency before they try to help students who might be at a higher fluency level than themselves. One suggestion was for each department to figure out what skills their students need and take care of advising and providing opportunities for improving IT fluency for their own majors. It was pointed out that the IT fluency advisor model we are advocating incorporates the idea that departments would advise their own majors but the advising of undeclared students still needs to have a technology component. This two-tiered model (general education advising followed by advising in the major) is still a problem for liberal arts faculty. It appears that some departments are not clear exactly what level of fluency their graduates will need. Word processing and an active user's level of information literacy seem to be enough. We may have to spell out the minimum level in the rubric that a graduate in a liberal arts discipline must achieve. This is not so much a problem in Math, science, or the professional disciplines.

The problem of advising beginning students in their general education program is being studied as part of the long range strategic plan. There is a recommendation in this plan for significant advisor training. Incorporated in that training would be strategies for helping students self assess their IT fluency level and develop their own plan to reach a target level before graduation.

We agreed that the efforts to improve faculty IT fluency must proceed apace with the effort to help students assess and improve their own fluency. We cannot afford to wait until faculty are fully fluent because the students we teach today must be prepared to enter a technological world.

One of the Science departments questioned the need for this effort. They have found that students they encounter, both majors and general education students (Saint Mary's has a one-year Science requirement of all students), are already sufficiently fluent in information technology to be successful after graduation. This department recommended that we survey the graduating seniors before doing unnecessary work. The end-of-semester crunch has made it impossible to prepare and administer such a survey. Perhaps we can assess the fluency level of the incoming first-year students and reconsider the fluency program on the basis of this assessment.

Conclusion

This report has outlined the approach to IT fluency assessment taken at Saint Mary's over the past year. It has detailed the insights and problems we encountered along the way and includes the documents we produced. Although we have had to scale back our plans several time, we still intend to spend time with each incoming student in the first few weeks next fall, helping them work through the rubric as part of their four-week introduction-to- technology course. We intend to make this information available to their academic advisors, along with the resource list we prepare over the summer from the faculty survey results, and to reassess the program after the intro course is completed.

In the present political environment at the college, we must make sure that we do not raise any faculty fears by appearing to act without proper clearance. One fear would be that we are trying to impose the rubric and the counseling of students in its use on the departments. It is not clear whether the rubric and its supporting documents should go through the curriculum committee. We could probably offer to submit it, but, since there is no credit or graduation requirement associated with the rubric at the current time, the curriculum committee would probably decide that approval of the rubric is outside its jurisdiction.

Other schools (e.g., George Mason University) have taken a much more agressive approach to the fluency problem. They have implemented Technology Across the Curriculum initiatives. We are hopeful that our low-key approach using student self-assessment and encouraging advisors to help students set and reach fluency goals, will achieve our stated goals. We understand that faculty may not be able to function effectively as advisors without explicit guidelines for using the rubric. We hope that these guidelines arise from our experience next fall.

Bibliography

[1] "Being Fluent with Information Technology," Committee on Information Technology Literacy, National Research Council, National Academy Press, Washington, D.C., 1999.

www.nap.edu/catalog/6482.html

[2] "Information Literacy Competency Standards for Higher Education," Association of College and Research Libraries, Chicago, IL, 2000. www.ala.org/acrl/ilcomstan.html

[3] "Grid for the Fluency in Information Technology Rubric," TLTR Task Force on Student IT Fluency, Saint Mary's College, Notre Dame, IN, 2001.

www.saintmarys.edu/~psmith/fluency.html

[4] "Information Technology Goals for Liberal Arts Students," by Ann Holisky, Associate Dean for Academic Programs, College of Arts and Sciences, George Mason University.

cas.gmu.edu/tac/docs/it_goals.html

Appendix 1









































Appendix 2

Survey Form

Name _____________________________

Department_________________________

Semester ___________________________

Course Name Course Number IT Identifier Code Representative Assignment Comment

Appendix 3

IT Fluency Task Force

Position Computer Skill Used
President, Student Diversity Board Adobe Photshop, Set up listserv, Word, Excel
Technology Com. Photoshop, Dreamweaver, Flash, MS Visual Studio, Office, Acrobat
Student Activities Board Website Adobe, Photoshop

Publicity Various Packages, Word, Excel, listservs

Secretary/Yearbook Managing Editor Page Maker, Dreamweaver, photoshop, MS Office
Student Academic Council Excel, Word (MS Office), not sure from publicity-will ask.
Student Body President Excel, Word, Adobe Page Mill, Photo Shop
Treasurer Excel, Word, Power Point, Pagemaker
Elections Word, E-mail (e.g., to facilitate abroad-student voting)
Campus Communications Commissioner Microsoft Newsletter Wizard
Residence Hall Association ClipArt Gallery, Corel Printhouse, Explorer and Netscape, Excel, Microsoft Publisher, MSAccess, Powerpoint, Microsoft Creative Writer, ClarisWorks, Printshop
Student Newspaper Use of MAC computers, Quark page design/ layout/publishing, photoshop.
Student Judicial Board Word, Excel, list serv.
Class of 2003 Officers PageMaker 6.5, Photoshop 6.0
Class of 2002 Officers Word, e-mail
Counseling & Career Development Center Adobe Pagemill, Pagemaker & Acrobat, JobDirect System (an on-line database of companies & jobs/internships). Alumnae Resource Network, Word Perfect, Word, MS Access, PowerPoint, Netscape & Internet Explorer, e-mail.
Athletics Word, Wordperfect, Excel, Photo Shop, Newsletter wizard, Adobe, pagemill, NCAA Stat programs, Microsoft Greetings

Appendix 4

Glossary of IT Identifiers

The following list of categories was adapted from a report by Dee Ann Holisky, Associate Dean for Academic Programs, College of Arts and Sciences, George Mason University. Each category has a wide range of complexity from elementary to advanced skills. If your course incorporates one or more of the categories at any level, please write the category letter in the correct column of the response form and briefly describe a representative course assignment that illustrates student accomplishment in that category. If no category applies, but your course does help students become more fluent with technology, put an X in the Category column and describe the technology component in the comments column. Note that we are asking you to enter a course on the survey form if it includes at least one of the descriptors in a category.

There are a couple of quotes from the introduction of Holinsky's report that help set the stage for our efforts. George Mason has a Technology Across the Curriculum (TAC) program which tracks where technology is incorporated into the curriculum. Although Saint Mary's has no formal TAC program, the TLTR is attempting to develop an informal data base of educational experiences which will help students improve their IT fluency and meet the fluency goals they are setting for themselves.

"Because we are not a training school, but an institution of higher education, the focus of this program is not on the teaching of technology skills or software packages, per se, but on identifying and developing those information technology skills that will enhance the learning objectives of specific courses. ... Moreover, as valuable as the skills identified below are, we believe that our most important job vis a vis IT is to develop in our students a conceptual understanding. It is this understanding which will best prepare them for a future in which the only certain thing is rapid technology change.

"Finally, though a list presents technology skills as discrete and unconnected, the ultimate goal is to help our students to integrate these skills. When confronted by complex problems, they should be able to choose the most appropriate technology tools to solve them." 1

A. Engaging in electronic collaboration.

This means using email (with file attachments) and listserves, understanding interaction between different modes of electronic communication, collecting material from a variety of electronic sources into a single document, collaborative writing, completion of a complicated group project, and understanding issues in the sharing of knowledge.

B. Using and creating structured electronic documents.

This means word processing (formatting and editing text, templates, styles, mail merge), web authoring using an editor (e.g. Dreamweaver) and understanding the uses of hypertext. It could mean knowledge of desktop publishing, image composition, advanced graphics, HTML, SGML, XML, electronic document structure, platform-independent programming, Web site management, server-side scripting.

C. Student technology-enhanced presentations.

This means employing basic Web skills as above or using basic features of presentation software, smooth use of technology and adaptation of technology design and display to audience. It could mean using high-end features of presentation software packages (custom templates, animation effects, multimedia, exporting); linking to other programs (e.g. spreadsheet); insertion of web links and management of real-time web access during presentations, video production.

D. Using appropriate electronic tools for research and evaluation.

This means developing appropriate search strategies (appropriate keywords, topics, resources), conducting Web searches, using different searching techniques (e.g. Boolean), evaluating sources using citations and abstracts, using on-line catalogs, news outlets, digital archives, conducting Web site evaluation, using bibliographic databases and standards, using discipline-specific databases and information services (JSTOR).

E. Using spreadsheets to manage information.

This means organizing data in worksheets (formatting ranges, columns, rows; multiple worksheets), formulas, column totals, lock columns, absolute/relative cell addressing, import/export data, simple graphing (pie chart, line graph, histogram, labels), understanding appropriate applications of spreadsheets. It could mean using macros, pivot tables, filters, statistical functions, logical functions, Visual Basic programming, interfacing with other applications.

F. Using databases to manage information.

This means setting up tables (define fields, add labels), editing records, conducting queries (sort/filter data), constructing forms or reports, understanding appropriate applications of databases. It could mean defining relationships between tables, creating complex queries, producing advanced multi-level reports, doing advanced forms, Visual Basic macros, basic SQL programming, and knowledge of data base management systems.

G. Using electronic tools for analyzing quantitative and qualitative data.

This means data entry and definition (read and describe variables and values; data verification or quality control), format output, source code control, reliability, validity, missing value analysis, descriptive statistics, plotting and graphing (pie chart, histogram, linear, scatter plot), formulating problems with an understanding of the relevant mathematical concepts. It could mean being able to use advanced statistical techniques and simulation theory.

H. Familiarity with major legal, ethical, and security issues in information technology.

This means familiarity with privacy issues, copyright and liabilities, netiquette, hacking, hoaxes and rogue programs (viruses, worms, Trojan Horses), open source versus proprietary standards.

I. Working knowledge of IT platforms.

This means familiarity with hardware, software installation, troubleshooting, operating systems (Windows, Mac OS, UNIX, VMS), browser-based graphical user interfaces, networks, file storage and directory structures.

J. Familiarity with the societal impact and limitations of technology

This means familiarity with the role knowledge workers play in society, knowing when and when not to use technology to tackle a social problem.

K.Proficiency using discipline specific technology.

Several disciplines have specific technology requirements different from the ones mentioned above. Students in these disciplines need to learn how to use these technologies and to interpret the results correctly.

X. Category not included above.

1"Information Technology Goals for Liberal Arts Students" by Ann Holisky, Associate Dean for Academic Programs, College of Arts and Sciences, George Mason University,

http://cas.gmu.edu/tac/docs/it_goals.html

 
 
Home - 2002 Conference - Proceedings - Newsletters - ASCUE-L Listserv - About ASCUE
©2001-2002 ASCUE, Inc.
email: clsmith@depauw.edu
http://www.ascue.org
Latest update: 3-nov-01