Onderzoeksmethoden 2/het werk/2010-11/Group07

Uit Werkplaats
Ga naar: navigatie, zoeken

Group Members Evi Antonopoulou, Benjamin Mader, Thalia Oikonomou, Danny Romme

Introduction

The earliest computers were mainframes that lacked any form of operating system. Each user had sole use of the machine and would arrive armed with program and data, often on punched paper cards and magnetic or paper tape. The program would be loaded into the machine, and the machine would be set to work until the program completed or crashed. Programs could generally be debugged via a control panel using toggle switches and panel lights. Symbolic languages, assemblers, and compilers were developed for programmers to translate symbolic program-code into binary code that previously would have been hand-encoded. Later machines came with libraries of support code on punched cards or magnetic tape, which would be linked to the user's program to assist in operations such as input and output. This was the genesis of the modern-day operating system (OS).


As machines became more powerful, the time to run programs diminished and the time to hand off the equipment to the next user became very large by comparison. Run queues evolved from a literal queue of people at the door, to a heap of media on a jobs-waiting table, or batches of punch-cards stacked one on top of the other in the reader, until the machine itself was able to select and sequence which magnetic tape drives were on line. Automated monitoring was needed not just for CPU usage but for counting pages printed, cards punched, cards read, disk storage used and for signaling when operator intervention was required by jobs such as changing magnetic tapes. Security features were added to operating systems to record audit trails of which programs were accessing which files and to prevent access to a production payroll file by an engineering program, for example.


All these features were building up towards the repertoire of a fully capable operating system. An underlying program offering basic hardware-management, software-scheduling and resource-monitoring may seem a remote ancestor to the user-oriented OS's of the personal computing era. But there has been a shift in the meaning of OS. Just as early automobiles lacked speedometers, radios, and air-conditioners which later became standard, more and more optional software features became standard features in every OS package. This has lead to the perception of an OS as a complete user-system with an integrated graphical user interface, utilities, some applications such as text editors and file managers, and configuration tools. 1


The number of different OS's developed in the past 50 years is almost endless 2, with some more successful than others. Today there are three major families of operating systems to choose from for personal computers: Windows, Mac OS, and Linux. These three families of OS's will be our main focus for the research.

Problem statement

So how does one decide which one of all those OS's is the best? Or at least best suited for his or her everyday tasks? The first step could be to ask family and friends on their experience with OS's. While they will definitely provide an honest answer, you cannot be sure if they have taken all aspects of the suitability into account. Another option could be to try out all the OS's yourself. This might be a good idea if you want to investigate the matter in-depth, but it is really time consuming and probably will cost you a small fortune. The fastest and least expensive option would be to scour through reviews on technology websites throughout the World Wide Web. But, as many of these reviews are written by biased critics, it is hard to make an objective choice.


To enable an objective selection of an OS and reduce the time spent on making that selection, we are going to find out what arguments are used by technology websites to answer the question: Which Operating System is the best?


We will take a look at reviews of OS's from different technology websites, analyse their arguments and try to find out if there is consensus regarding the pros and cons of the different OS's. The answer on this will be an extensive overview of their scores on a variety of subjects.

Method

In this study, we are comparing reviews on operating systems from different technology websites, using text analysis as main method. Because of the vast amount of different OS's we chose to narrow our research down to the newest versions of today's top-3 most-used OS's. A short introduction to those three operating systems is provided next.


Windows 7 is the latest release of Microsoft Windows, a series of operating systems produced by Microsoft for use on personal computers. Windows 7 was released to manufacturing on July 22, 2009 and reached general retail availability on October 22, 2009, less than three years after the release of its predecessor Windows Vista. Unlike its predecessor, though, which introduced a large number of new features, Windows 7 was intended to be a more focused, incremental upgrade to the Windows line, with the goal of being compatible with applications and hardware which Windows Vista was not at the time. Presentations given by Microsoft in 2008 focused on multi-touch support, a redesigned Windows shell with a new task bar, a home networking system, and performance improvements. 3


Mac OS X Snow Leopard (version 10.6) is the seventh and current major release of Mac OS X, Apple's desktop and server operating system for Macintosh computers. Snow Leopard was publicly unveiled on June 8, 2009 at the Apple Worldwide Developers Conference. On August 28, 2009, it was released worldwide, and was made available for purchase from Apple's website and its retail stores at the price of US$29 for a single-user license. As a result of the low price, initial sales of Snow Leopard were significantly higher than that of its predecessors. The release of Snow Leopard came nearly two years after the introduction of Mac OS X Leopard (version 10.5), the second longest time span between successive Mac OS X releases. Unlike previous versions of Mac OS X, the goals with Snow Leopard were improved performance, greater efficiency and the reduction of its overall memory footprint. 4


Ubuntu is a fork of the Debian project's codebase. The original aim of the Ubuntu team was to create an easy-to-use Linux desktop with new releases scheduled on a predictable six-month basis, resulting in a more frequently updated system. Ubuntu's first release was on October 20, 2004. Since then, Canonical has released new versions of Ubuntu every six months with commitment to support each release for eighteen months by providing security fixes, patches to critical bugs and minor updates to programs. It was decided that every fourth release, issued on a two-year basis, would receive long-term support (LTS). LTS releases are supported for three years on desktop PC's. The latest LTS release is Ubuntu 10.04 (Lucid Lynx), released on April 29, 2010. The distribution emphasizes the new importance of web services and social networking with integrated interfaces for posting to sites like Facebook and Twitter, complementing the IM and email integration already in Ubuntu. 5


Approach

We selected reviews from reliable technology websites. From these reviews we selected the arguments that are against or in favour of the three operating systems, so that we have a collection of arguments from different sources. For each argument we defined the category in which it should be included. Then we accumulated these arguments and categories on a table. For each table row, we represent the argument, the category it falls into, its weight and the source website in which it was found.

We decided to copy the original argument, as it was mentioned in the review, so that data would not be altered or misinterpreted before even analysing it. We judged each argument as being negative, positive or neutral and assigned a weight indicating its importance. After we weighed each argument, we computed the average weights per category for each OS review. Then we compared the average category weights of all reviews concerning one particular operating system. For each OS and category we computed the average weight and the standard deviation from this average, by the individual weights assigned to the category by each review of the same OS.

Then we assigned three different values of consensus (High, Medium, Low) to these standard deviations. Lower standard deviation values indicate higher consensus between the reviews. We then analysed these values to find out how much consensus there is between the different reviews for one operating system.


Conceptual model

We created a conceptual model, to give a clearer view of the interdependencies of the data we collected in our research.

Conceptual model V3.1.png

Data gathering

We analysed 13 reviews from well known technology websites, evenly distributed among our three operating systems. Then we derived the arguments reading the reviews and we identified in which category each argument could be best included. Each argument was assigned a source id, indicating the website review in which it was mentioned, a category that it falls into and a relative weight in this category, indicating the importance of the specific argument for the common user. It was difficult to define the method for assigning relative weights to arguments. Many different options were available. At first we thought of having a weight per category, which would then be inherited to the argument. But this would not represent the actual significance of the argument, as it would be very generalized. Another option was to have the assigned weight computed as a combination (e.g. average) of weight per argument and per user category (examples of user categories would be hardware developers, application programmers, ordinary users, system managers etc.). This, would add complexity in our conceptual model, and would not be necessarily useful for deriving a safe conclusion. Furthermore, differentiation among user categories was not possible since the texts that we analysed were mainly reviewing operating systems according to common users' criteria.


Finally we agreed to have a common way of assigning weight to arguments, based on the importance this specific argument seems to have in its respective review and some basic common criteria that are established according to their contribution in completing or performing everyday tasks. Furthermore, in assigning the weight we also took in consideration how often an argument is mentioned in the reviews we analysed. Weight values assigned to arguments are described below.


  • 1  : if the argument refers to a characteristic significantly helpful for daily tasks
  • 0.5  : if the argument refers to a characteristic helpful for daily tasks
  • 0  : if the argument is neutral
  • -0.5 : if the argument refers to a characteristic hardening completion of daily tasks
  • -1  : if the argument refers to a characteristic significantly hardening completion of daily tasks


Below we include the reviews that we analysed, and the ID we will use to refer to them from now on:

Data structuring

Variables

We have decided to use 10 categories to classify the arguments used in the reviews. We chose these categories based on reference document 6.

Ten categories to rate Operating Systems on
# Categories Description
1 Reliability A measure of how consistently the OS runs applications and services
2 Security The ability of an OS to protect itself from malicious intrusions
3 Scalability The ability to adept to a growing environment without having to prepay for overcapacity
4 Manageability The effort it takes to keep the OS up and running
5 Compatibility A measure of how well an OS complies with certain standards
6 Adaptability A measure of how well an OS can interact with other systems or programmes
7 Affordability The costs of acquiring and usage of the particular OS
8 Ease of use The ease at which the OS installs and at which everyday tasks can be performed
9 Performance A measure of how well the OS performs certain tasks, also considering the machine's capabilities
10 Bundled Applications Applications which are not a core part of the OS but are installed by default.

Data representation

Data format: 


<argument>								
<OS></OS>			Operating system
<id></id>			Unique identifier
<src></src>			Source article
<type></type>			Type of the argument (positive/negative/neutral)
<weight></weight>		Importance of the argument
<category></category>		Category it falls under
<statement></statement>	Text of the argument
</argument>
The gathered data presented in the above data format

Data analysis

To determine whether there is consensus among the analyzed reviews we used the standard deviation of the average category weights. The maximum possible standard deviation is 1.41 in our case (average weights -1 and 1).In order to define 3 areas of consensus (high,medium,low ) we used the following bounds (1.41/3)*1= 0.47 ,(1.41/3)*2= 0.94,(1.41/3)*3= 1,41 : Following this approach the consensus among arguments of different reviews for a specific category and OS was evaluated as :

  • high  : [0,0.47]
  • medium  : (0.47,0.94]
  • low  : (0.94,1.41]

Consensus

Please note that categories, for which no arguments were present in a particular review, will be assigned the weight "0" fo this article. We assume that the review regards this category as neutral, or else it would have argued positively or negatively for the OS in this category.

Using the average standard deviation OS, we can make a statement about how consistent the different reviews view the OS in question. Please note that only categories for which arguments were present in at least 1 review will be considered while calculating the average stddev. For example, category 6 is not represented in any average standard deviation because not 1 review contained arguments that belong in this category.

Consensus between Mac OSX Reviews
Category S1 S6 S4 S9 Average Standard Deviation Consensus
1 0 0 0 0.5 0.13 0.25 High
2 0.5 0 -1 0.5 0.0 0.71 Medium
3 0 0 1 0.5 0.38 0.48 Medium
4 0 0 0.5 0.75 0.31 0.38 High
5 1 0.17 0 -0.17 0.25 0.52 Medium
6 0 0 0 0 0 0 ---
7 0 0.5 0 0 0.13 0.25 High
8 0 0.27 0.36 0.5 0.28 0.21 High
9 0.17 1 0.64 0.83 0.66 0.36 High
10 0 0 0.5 0.5 0.25 0.29 High

Average Weight Mac OSX: 0,26

Average Standard Deviation Mac OSX: 0,38

Consensus between Ubuntu 10.04 LTS Reviews
Category S2 S8 S12 S13 Average Standard Deviation Consensus
1 0 0 0 0 0 0 ---
2 0 0 0 0 0 0 ---
3 0 0 0 0 0 0 ---
4 0 0.5 0 0 0.13 0.25 High
5 0 0 0 -0.5 -0.13 0.25 High
6 0 0 0 0 0 0 ---
7 1 1 0 0 0.50 0.58 Medium
8 0.5 0.5 0.5 0.64 0.54 0.07 High
9 0.83 0 0 -0.5 0.08 0.55 Medium
10 0.5 0.67 0.13 0.25 0.39 0.24 High

Average Weight Ubuntu 10.04 LTS: 0,25

Average Standard Deviation Ubuntu 10.04 LTS: 0,32

Consensus between Windows 7 Reviews
Category S7 S3 S9 S10 S11 S5 Average Standard Deviation Consensus
1 -0,5 0 0 0 0 0 -0,08 0,20 High
2 0 -0,5 0 0 -0,5 0,5 -0,08 0,38 High
3 -0,5 0 0 0 0 0 -0,08 0,20 High
4 0,5 0 -0,5 0,5 -0,5 0 0 0,45 High
5 0 0 0 0,5 0,5 0 0,17 0,26 High
6 0 0 0 0 0 0 0 0 ---
7 0,5 0 0 0 0 0 0,08 0,20 High
8 0,06 0,29 -0,5 0,5 0,17 0,5 0,17 0,37 High
9 0,5 0 0 0 0,5 0,75 0,29 0,33 High
10 -0,5 0,5 0 0 -0,36 0,5 0,02 0,42 High

Average Weight Windows 7: 0,05

Average Standard Deviation Windows 7: 0,31

Analysis

At first we want to take a look at the overall rating of the operating systems by the analyzed reviews.

If we take a look at the average standard deviation of the 3 operating systems, then it is easy to see that all 3 are quite similar. While Ubuntu and Windows receive almost identical values (0,32 and 0,31), the reviews of OSX seem to agree a little with a standard deviation of 0,38. Overall, the differences between the 3 values are so small that we can conclude that the consensus between different reviews is high, which means that they pretty much agree on the advantages and weak points of the operating systems in question.

Now that we know that the average standard deviation between the reviews is low enough, we can examine the average weight with confidence. Once again there are 2 OSs which are rated almost the same. These two are OSX and Ubuntu, with an average rating of 0,26 and 0,25. These two OSs are rated positively by the reviews, whereas Windows falls behind significantly with an average rating of 0,05, which just barely falls in the positive area.

A further interesting finding is that the only OS with all positive average category weights is Mac OSX. While OSX sports a flawless record, Windows has 3 negative categories (Reliability, Security, Scalability) and Ubuntu 1 (Compatibility). This seems logical, as Reliability and Security were always considered weak points of Windows systems by many people, while Ubuntu, because of its non-commercial nature, can not support as many hardware combinations as Windows.

Let`s take a look at the OSs scores (average weights) in the different categories. The sequence in which the OSs are listed indicates the ranking in the category in question:


1 - Reliability

OSX: 0,13 (stddev: 0,25)

Windows: -0,08 (stddev: 0,20)

Ubuntu: No arguments in reviews

Unfortunately there were no reliability related arguments in the Ubuntu reviews, so we can only compare OSX and Windows in this category. The standard deviations are quite similar, but the weights are very different. While OSX receives 0,13, which is slightly positive, Windows receives -0,8 which is slightly negative.


2 - Security

OSX: 0 (stddev: 0,71)

Windows: -0,08 (stddev: 0,38)

Ubuntu: No arguments in reviews

In this category, OSX and Windows are pretty close. While OSX has a neutral score of 0, the standard deviation is quite high, which means that there was no strong consensus between the reviews in this category. Windows on the other hand has a slightly negative weight of -0,08 but with a lower standard deviation. This leads us to the conclusion that these 2 OSs are on par in this category.


3 - Scalability

OSX: 0,38 (stddev: 0,48)

Windows: -0,08 (stddev: 0,20)

Ubuntu: No arguments in reviews

Once again OSX gets the highest average weight, although once again accompanied by a higher standard deviation. Windows is once again slightly negative while there were no arguments concerning ubuntu.


4 - Managability

OSX: 0,31 (stddev: 0,38)

Ubuntu: 0,13 (stddev: 0,25)

Windows: 0 (stddev: 0,45)

In this category, OSX once again gets the highest average weight, once again accompanied a high standard deviation. While Ubuntu receives a slightly positive weight, Windows only receives a neutral weight.


5 - Compatibility

OSX: 0,25 (stddev: 0,52)

Windows: 0,17 (stddev: 0,26)

Ubuntu: -0,13 (stddev: 0,25)

In this category, OSX once again receives the highest average weight, this time also accompanied by the highest standard deviation. Windows and Ubuntu both have small standard deviations while Windows receives a positive weight and ubuntu a slightly negative one. This is the only category where Ubuntu receives a negative weight.


6 - Adaptability

OSX: No arguments in reviews

Ubuntu: No arguments in reviews

Windows: No arguments in reviews

There were no arguments which would have fallen into this category, even though we expected otherwise. Therefore this category did not have influence on our conclusion.


7 - Affordability

Ubuntu: 0,50 (stddev: 0,58)

OSX: 0,13 (stddev: 0,25)

Windows: 0,8 (stddev: 0,20)

Ubuntu ranks highest in this category, this comes as no surprise because Ubuntu is a free download.. It is followed by Mac OSX with a € 25,- price for the full retail version, and Windows with its plethora of versions and price points reaching as high as $ 299,-. Once again OSX has the highest standard deviation.


8 - Ease of Use

Ubuntu: 0,54 (stddev: 0,07)

OSX: 0,28 (stddev: 0,21)

Windows: 0,17 (stddev: 0,37)

Surprisingly, the free alternative Ubuntu 10.04 LTS scores the highest average weight in this category. Please also note the very small standard deviation which indicates incredible strong consensus between the different reviews. OSX comes in as second and Windows as third. Both have positive weights but considerably higher standard deviations than Ubuntu.


9 - Performance

OSX: 0,66 (stddev: 0,36)

Windows: 0,29 (stddev: 0,33)

Ubuntu: 0,08 (stddev: 0,55)

OSX takes the lead again, followed by the also positive weighed Windows 7 and Ubuntu, The standard deviations of Windows and OSX are about the same and Ubuntu shows a considerably higher value.


10 - Bundled Applications

Ubuntu: 0,39 (stddev: 0,24)

OSX: 0,25 (stddev: 0,29)

Windows: 0,02 (stddev: 0,42)

Ubuntu takes the lead again, closely followed by OSX. These two OSs are rated clearly positive while Windows`s average weight is barely positive. Also note the higher standard deviation of Windows.

Conclusion

We wanted to find out if there is a consensus between reviews when evaluating operating systems.

After conducting the research and analysing the data, we can say that the consensus between reviews for each of the 3 OS's is quite high (0,31 - 0,38). The values are so close together that it is hard to conclude which OS has the most consensus in its reviews, so we can say that the review consensus is high for all OS's.

Now that we know that standard deviations are consistently low, we can trust the average weights of the reviews to reflect a common attitude. Now we can ask the question:Which OS is the best?

According to the reviews we analysed, Mac OS X is the best OS out of the three. This is based on the fact that OS X has the highest overall average category weight (0,26), the highest average weight in 6 categories (Reliability, Security, Scalability, Manageability, Compatibility, Performance) (Ubuntu: 3; Windows: 0). It should be noted that the standard deviation of weights for OS X seems to follow a certain trend: It is almost always higher for OSX than for the other two systems. We conclude from this that OS X is also the most controversial OS out of the three.

Ubuntu 10.04 LTS comes in as a close second. Its average category weight is only slightly lower than the one of OS X (0,25 - 0,26) but its standard deviation is slightly better (lower) than the one of OS X (0,32 - 0,38), indicating more consensus among the Ubuntu reviews. The most important reason for Ubuntu only reaching the second place is, that it could only dominate, weight wise, in three categories (Affordability, Ease of Use and Bundled Applications) while OS X did in 6. Ubuntu also has a negative weight in one category (Compatibility).

Last but not least, Windows 7. It has the by far lowest average category weight (0,05) and a standard deviation on par with (even slightly better than) Ubuntu. So while the other OS's score clearly positive average weights, Windows`s average weight only slightly positively deviates from neutral. Also, Windows did not dominate a single category and got 3 negative category weights (Reliability, Security, Scalability) as opposed to 1 for Ubuntu and 0 for OS X.

Reflection

Conceptual

For the conceptual model we chose to use ORM (Object Role Modeling), a powerful method for designing database models at the conceptual level. Formulating the conceptual model was the first part of our analysis, after the problem had been defined. The very initial model was extended by some further concepts we came up with while doing our research, but it then remained unchanged. It reflects the basic concepts that have guided our research and on which we focused, such as technology website, operating system, argument, category and importance weight. The concepts argument, site (or source), Operating System, category and weight are used in the tagging scheme. We must point out that our decisions regarding the model were based on our effort to keep it as simple as possible, according to the remarks and hints given during the lectures.

Method

First we define the conceptual model reflecting the basic concepts related to the defined problem statement. The initial conceptual model included only some basic concepts (technology website, operating system, argument). However, after further discussion we decided to extend it so as to include some additional concepts (category, importance weight). This conceptual model is the one on which we finally based our content analysis. Since we use text analysis method, a tagging scheme for representing gathered information in a structured way was applied. The tagnames reflect the concepts found in the ORM.

Thirteen reviews from reliable technology sites were used as the sources of the arguments to be analyzed. We divided the sources and skimmed them, indicating and noting the arguments referred in each one of them. Then we agree on categories that we would use so as to classify the arguments depending on their functionality and usage. Thus we came up with ten categories as referred and explained in section “data structuring”. Then we had to weight the arguments so as to define their importance and contribution on the final conclusion. For this process we thought of many different alternatives. At first the predominant one was to weight each argument per user category. Though, the nature of our sources (technology websites reviews that express professionals’ opinions about the advantages /disadvantages of operating systems for common tasks and users) made this approach rather improper. Thus we agreed on having a more subjective weighting method consisting of the importance the argument had in the review (as expressed by the author), its frequency within the analyzed review as well as our personal opinion about its significance. Having gathered, classified and weighted the arguments we structured them, using the tagging schemed and proceeded with their further analysis.

We created excel files in which initially we calculated the average weight for each review and category. Then for each operating system we calculated the average and the standard deviation for each category. Using the maximum standard deviation we defined three areas of consensus as described in section “Data Analysis». Each category per operating system fell into one of the areas of consensus. This result in combination with the average weight was used as an indicator of the importance of each operating system in each category.


Summary

Having followed the lectures of the course ‘Research Methods’, we had to complete the practical assignment consisting of a research based on the method of ‘Content Analysis’. From the beginning we figured out that we had to use a structured approach. We defined the topic that we would analyze and the sources that we would base our analysis on. Afterwards, all team members worked together on these sources. A more detailed description of the approach we followed can be found in the ‘Reflection’ section. We should mention that while working on the topic we did not encounter serious problems, apart from the ones related to the weighting method (also described in the ‘Data gathering’ section). The result is a clearly structured research while the process is easy to be followed through, from beginning to end.

We managed to get clear, unambiguous results which mostly reflect our expectations while also providing some interesting and unexpected findings on the topic.

References

[1] History of Operation Systems

[2] Time Line of Operating Systems

[3] Microsoft Windows 7

[4] Mac OS X Snow Leopard

[5] Canonical Ubuntu Lucid Lynx

[6] Evaluation criteria

Overgenomen van "https://lab.cs.ru.nl/algemeen/index.php?title=Onderzoeksmethoden_2/het_werk/2010-11/Group07&oldid=129647"