In the beautiful city of Chicago, thousands of practitioners and academics within Industrial/Organizational Psychology gathered for SIOP 2024. A number of trends dominated the conference including:
- AI’s unwillingness to leave – I/O psychologists must understand it fully before we get left behind
- A need for more interconnectedness between academic researchers and practitioners on the ground in a dynamic and ever-shifting world
- Bridging the gap between the C-Suite/senior leadership and I/O expertise
- Deep dish pizza is not great – it is not authentic Chicago pizza (the author of this section, Rohan, is from Chicago and is allowed to say this)
While these trends are important and prevalent, the E. Rogers Associates team attended a number of interesting sessions with other unique insights.
One such session, I-O Psychology in Sports Contexts: Perspectives From Experts Bridging the Fields, provided general information of using I/O psychology within a sports context and helped through sharing perspectives of experts in these areas to the overlap with areas of overlap within sports and I/O. Mainly in Witt et al. (2024)’s panel, there was discussion of how the panelists were able to enter into the realm of sports organizations. Notably, two of the panel members were former professional athletes. The core of the discussion was how they are able to leverage their knowledge of I/O psychology within their day to day of their roles. While their experiences varied, the key areas of discussion surrounded performance management, employee well-being and coaching for success. With this session in mind, we will present our takeaways from SIOP 2024 using the context of a sports related scenario:
Scenario: The Plight of the Washington Commanders
Imagine you are a professional football team that has experienced a tumultuous 20 years of inconsistency. This is in part due to performance on the field, but also poor management of the organization, ultimately leading to a lackluster fanbase engagement. You have experienced some good times, but mainly have embraced mediocrity. A historic franchise, which was once referred to as one of the premier in the league, is now consistently the joke of the league due to a questionable majority owner. This is the state of the Washington Commanders going into the 2022 NFL Season.
As with any offseason, questions loomed regarding player and coaching personnel decisions. However, everything shifted when the now former owner was pushed to sell the franchise due to reports of harassment, misconduct, financial impropriety, and more. A new era had begun in D.C., with an ownership group laser-focused on shifting the narrative of the franchise and creating hope for an organization and city yearning for its glory days again.
As a new leadership group has come in, they desired to reset expectations and the vision of the organization. They have leaned into adopting strategies from an I/O perspective and are wanting to understand what strategies to implement within their organizational practices.
Revisiting the Strategy Behind 360’s
When looking at the Washington Commanders, it is imperative to understand the historical background of the franchise in addition to the ownership group. Establishing stakeholder analyses will help engage the leadership team in addition to discovering their goals and desires for the organization going forward. Using the knowledge from the session Breathing New Life Into Old 360 Feedback Systems, the main idea, according to Church et al. (2024), is to shift away from the traditional mindset of performance appraisal to more of a modern 360 feedback system.
This strategy to performance management generally provides feedback from multiple levels in the organizations, lending to unique
and different perspectives for individuals to grow, develop and learn from. Typically, with 360 systems, there is a focus on assessments for individual contributors and managers to take in order to identify areas for improvement. In addition to those, there is an emphasis on the people managers or leaders to discover which areas or skills to build for leadership development.
The ideas from this session embodied what we know of traditional leadership development in regard to 360 systems, while also shedding light on a new perspective which some notable organizations have been doing quite well. The specific ideas discussed in these panels focused on putting the emphasis on “soft” skills for leadership development as opposed to the standard measures for HiPo’s or leaders.
This was shown in the example of shifting the strategy for leadership from identifying skills such as critical thinking or communication as core skills for leaders towards softer skills such as empathy. This is due to the fact that the majority of the previously identified skills for leaders are usually inherent with HiPo’s and leaders, but the differences between individuals and variance was shown when using the framework of 360 to the softer skills. This was also to embody the executive leadership team’s characteristics and culture they wished to establish within those organizations.
As we look at this for the Commanders, we would start by educating within the organization to this new shift of mindset from leadership, but also will help spread the awareness of the changes coming. This is the beginning of building out a new culture within the organization for continuous learning, development, and more importantly feedback.
Shifting the Thinking that Goes Into Assessment Item Construction
Now that the Commanders’ leadership understands that the goals for their organizational assessments should embrace things such as the measurement of both soft skills and traditional KSA’s (Knowledge, Skills & Abilities), they are now able to focus on how these 
competencies can be measured. Leaning more into the findings from SIOP 2024, they discovered another way to create the assessment items. In their session, Advancing Organizational Science with Process Theory, Data, and Analytics, Chao et al. (2024) spoke about the idea of “process thinking”.
Before explaining this thinking, it is critical to first set the stage. Currently, our academic and professional work is dominated by “factor” or “construct” thinking. Basically – we look at measuring specific static factors (such as ‘sociability’ or ‘preference for critical thinking’) and then try and see how these factors correlate with each other. While this way of theorizing has clearly pushed our field further, it does not capture the full picture.
In contrast, process thinking is focused on the “affective, behavioral, cognitive and social actions of actors”. Research has shown more direct support for this kind of thinking rather than the more inferred results from construct thinking. Process thinking would ask how certain outcomes are reached. It is focused on action sequences and dynamic event sequences. Since we are talking about football, here is an analogy to illustrate the differences between the two types of thinking:
Construct thinking would look at the correlation between a quarterback’s accuracy and the amount of points the team has scored. In contrast, process thinking would look at the amount of time the quarterback takes before throwing, how far they throw, where and how the receiver catches the ball, how many yards the receiver runs, and how that leads to points scored.
Bringing it back to organizational assessments, the Commanders may wish to measure how their teams and leaders foster a culture of knowledge sharing. The Commanders’ front office will be full of functional experts in various roles such as college scouts, player development personnel, and even the coaches on the field. If they are to stay ahead of their competition, they must ensure the front office has a clear shared mental model. Using process thinking they may use items that measure specific mechanisms and actions such as “When I gain a new piece of information, I share it with my team” and “My supervisor provides me information from senior leadership”. Using this kind of thinking may seem like a subtle shift, but it may provide a more comprehensive picture on what needs to be measured.
Addressing Concerns of Missing Data with a Reliable but Forgotten Technique
Now this new thinking may lead to a large amount of items being
added to their assessments. The Commanders may be worried about participant exhaustion and incomplete surveys. They may be concerned that they will not get sufficient data and cannot make proper conclusions. Their first reaction may be to go back to a thinking that only asks a few questions. This will not burden the survey takers, but they risk not obtaining clear connections to identified outcomes. Another urge may be to eliminate entire areas or sections. However, this defeats the purpose of measuring the competencies that have been identified as important needs to the organization.
Another session, Planned missingness to reduce survey length: A sheep in wolf’s clothing, may offer a solution. In this session, Zhang et al. (2024) do not offer a new radical method, but reintroduce a traditional survey statistical technique known as “planned missingness”.
With planned missingness, assessment designers will only show a limited amount of the items within the full survey to each survey taker. Each participant will receive a set number of items, but the items they are faced with will be randomly selected for them. With this technique, the Commanders can ease concerns of survey incompleteness. It is critical to note that this technique is not appropriate in every situation, such as when only a limited number of survey takers are present – it would not be a useful technique within a small team. However, if used correctly such as through perhaps a large climate survey, the Commanders can gain more data with less burden.
The Surprisingly Diverging Results behind Scale Designs
While the E. Rogers Associates team were eager attendees of SIOP 2024, two of our teammates, Amar Bhatia and Rohan Ramesh, were fortunate to be eager presenters as well. The two of them, along with their colleagues of the “LAME” (Leadership And Measurement Experience) Lab, presented two posters that presented results from our studies on survey design:
The first poster session, The Impact of Response Format: Sliders Versus Radio Button Scales, discussed the differences between radio buttons and sliders. The findings of this research provided insight into the effectiveness of the survey design to the users, in addition to understanding in what context to use which in. Overall, the results suggest that a 7-point measurement scale in surveys with radio buttons will display the highest averages across the different levels of 
variables. Additionally, respondents were less likely to drop out in surveys with radio buttons as opposed to surveys with sliders.
The other poster session, Following up on the Impact of Verbiage and Scale Points on Scale Responses, looked at the differences between surveys that use a 5-point scale or a 7-point scale. Additionally, we looked at different types of survey labels within the 5-point scale options (e.g. one scale’s extreme positive option was labeled as “satisfied” whereas another scale’s extreme positive option was labeled as “extremely satisfied”). We did not find any meaningful differences between the 5-point scales with different labels, implying that survey designers should not spend as much time on the labels of the scale. What we did find was that the 7-point scale resulted in higher means and higher variances (meaning results were more varied) than any of the 5-point scales.
As we look to help build a better culture of feedback and development for the Commanders, understanding the differences between scale formats and selecting the one best for their situation can benefit the overall engagement in surveys being used for performance feedback, employee well-being, and even fanbase surveys. Are they worried about respondent attrition? Use a simple radio button scale format rather than a slider format. What if they are worried that respondents may think too much about specific sections and may be too indecisive? Use a 5-point scale as opposed to a 7-point scale since 7-point scales tend to have more variance.
Ultimately, when they understand the different types of survey types and choose the design best for their situation, the Commanders will have more accurate variable data. This will allow them to gather insights and drive greater effectiveness in strategies. This will help shape a pathway for the goals of the franchise going forward.
As our colleagues concluded their time at SIOP, the numerous connections and new knowledge which they brought will remain with them as they continue to navigate their careers. We believe these takeaways can help an organization like the Commanders form their new brand while also reshaping their culture. The I/O centered strategies discussed above will guide the ownership group of the franchise to foster a championship culture and bring Super Bowls back to the D.C. area.
References
Chao, G. T. (Chair), Braun, M. T., Grand, J. A., Kozlowski, S. W. J., & Kuljanin, G. (2024). Advancing organizational science with process theory, data, and analytics [Panel]. Society for Industrial and Organizational Psychology Annual Conference, Chicago, IL, United States.
Church, A. H. (Chair), & Seibert, J. (Discussant). (2024). Breathing new life into old 360s [Symposium]. Society for Industrial and Organizational Psychology Annual Conference, Chicago, IL, United States.
Shapiro, T., Hymowitz A., Njaka A., Mattis C., Cigaran, E., Ramesh, R.,(2024). Following up on the impact of verbiage and scale points on survey sesponses [Poster]. Society for Industrial and Organizational Psychology Annual Conference, Chicago, IL, United States.
Shapiro, T., Njaka, A., Hymowitz, A., Mattis, C., Cigaran, E., Ramesh, R., Bhatia, A. (2024). The impact of response format: Sliders versus radio button scales [Poster]. Society for Industrial and Organizational Psychology Annual Conference, Chicago, IL, United States.
Witt, L. A. (Chair), Waite, L., Shoenfelt, E. L., & Bernstein, A. F. (2024). I-O Psychology in Sport Contexts: Perspectives from Experts Bridging the Fields. Society for Industrial and Organizational Psychology Annual Conference, Chicago, IL, United States.
Zhang, C. & Sackett, P. R. (2024). Planned missingness to reduce survey length: A sheep in wolf’s clothing. [Symposioum] Society for Industrial and Organizational Psychology Annual Conference, Chicago, IL, United States.