The following transcript has been edited for length and readability. Listen to the entire discussion here on The Broadband Bunch. The Broadband Bunch is sponsored by ETI Software.
Brad Hine:
Hello everyone in broadband land. Welcome to another episode of the Broadband Bunch. I’m your host, Brad Hine, and we are on-site here at the UTC Annual Conference in beautiful Fort Lauderdale, Florida. Lee Ayers, Vice President of Engineering for Mid-Carolina Electric Cooperative is here with me today. Welcome to the bunch, Lee.
Lee Ayers:
Hey, thanks for having me, Brad. Look forward to our conversation.
Brad Hine:
Absolutely. I know that you’ve been very involved in UTC over your career.
You shared some great information with me in our discussion before this interview. Tell us a little bit about your background and how you got involved in engineering and then jumped on board in EC there.
Lee Ayers:
I’ll try to keep it concise because it’s certainly not short. I’ve worked for electric co-ops my entire life. I’m a second-generation electric cooperative employee. My dad retired from it several years ago. I’m a 1982 Clemson grad and have worked for electric co-ops since then, which if the math works out, it’s about 41 years.
I’ve been at Mid-Carolina for 37 years. My initial background was more of a power emphasis. But my entire career I’ve been attaching technology to the power system. I was doing smart grid way before they coined the name.
Because every time we’ve attached technology, the grid’s gotten better. It’s gotten smarter; and it’s a journey, not a destination. It’s not a definition. But I’ve been at it now at Mid-Carolina for 37 years.
I’ve also had an interest in communications and the communications facilities and technologies that are needed to strengthen the grid. And I think back to when I was first hired, we already had a supervisory system, a SCADA system that monitored our substations. We communicated with it over old analog lease lines from the phone company. And my boss said, “Your first goal is to get lease line outages to less than five days.”
I was like, “What?” And he goes, “Yeah.”
Well, a few years later we put in our own private communication system. A core thing to us here at UTC is why do utilities have private communications networks? Well, because we need them to work when the weather’s the worst. That’s when we have trouble with the power system. We need to be able to communicate.
So we put in an analog microwave system back then. It served us well until about 2002. Then we decided to put in our first automated metering infrastructure, AMI. The management team at the time came to me and said, “Project’s yours, and you need to figure out how to have it communicate over that analog microwave system.”
I said, “Hmm, that’s an easy answer, it won’t.”
It was obsolete for buying more channel parts for it 10 years ago because we bought it with limited expansion capability. And it was really a technology that was at the end of life when we bought it. So at that point, we had to go back to the telephone company world to lease some digital lines, which we did.
And as we got our AMI system rolled out over the next two-and-a-half years, we were already having trouble keeping those lines up. Now, meter reading at that time was really our first and foremost job was to get reading to make a bill once a month. So uptime wasn’t that important, but we were also still running our SCADA system over it. We had to transition it.
So this is on the falling edge of the dot-com bubble. It had burst and the fiber optic cable manufacturing world was begging for customers. We had a fiber optic manufacturer about a mile-and-a-half down the road from our headquarters that came to us and said, “Hey, now’s the time.”
We had looked at connecting our two offices about five years earlier, and the business case was just cost prohibitive. When they brought us the new numbers and some of the advancements in technology for electronics, we made a good business case out of it. So we put our first fiber in, in ’05. All we did was connect two offices and 20 substations, but I was always thinking ahead. Always having a vision, always, what’s the future going to look like? What might we need?
Even then I thought, “Hey if we get this in, we could serve the internet.”
And I mentioned that to the management team and they said, “Oh no, we’re not going to do that. We’re a power company. We don’t do that. We let the telephone company do that; we let the cable TV company do that.”
But it never left my mind. We rolled along the early 2010s to ’11, ’12, or ’13, somewhere in there. We changed out a bunch of our protective equipment down line known as reclosers, things that make the lights blink.
Brad Hine:
Yeah. Oh yeah.
Lee Ayers:
They go off when the fault gets online. They cut off and let it clear and come back on. That is the correct operation. But we’d changed them out to smart equipment that was capable of communication, and we began connecting. I had built that first fiber ring with that in mind and had access points co-located with the sites where we were changing the equipment.
We had about 65 or 70 of 250 electronic recloser locations on our SCADA system on fiber. And we were slowly adding about $250,000 worth of fiber optic plant a year. And the boss, the new CEO, and I were sitting down doing some strategic planning and strategic budgeting, and he said, “Well, how many more do we have to do?” And we did the math and we both calculated that we would probably be retired for 10 years by the time we could build out the rest of it at the pace we were going.
So we got to brainstorming and went through the thought processes and the questions of what we needed to serve. What would we like to serve? How far do we need to go into the electrical network with fiber? Ultimately, got to the how far can we go? And we said we can develop a system, design a system that we can go all the way to the meter on the side to house, on the side of the store, on the side of the office building. And that’s what we ultimately designed and deployed.
But the other big problem that came out of that was that’s going to be a lot of money.
Brad Hine:
Right. I was going to say, $250,000 is a lot of money, but not when you’re talking fiber networks annually.
Lee Ayers:
No, you need to multiply it by about 30. Yeah, add a zero and multiply it by three.
But we knew that we had co-op members that needed internet. They were unserved. We had participated in a funding opportunity in 2009 and ’10, the ARRA program, and we didn’t get the grants. But in doing that work, we’d established a need.
So co-ops, what are they? They’re people who have a common need and form a co-op to service that need. That’s an electric co-op or any other co-op. In the history of the electric co-ops back in the late 1930s and into the early 1940s, electric co-ops were formed to provide electric lights to the countryside and to the locations that had a low density of population that the for-profit, investor-owned utilities could not show adequate return on investment to satisfy their stockholders. So electric co-ops were born to provide lights.
Well, what’s fiber optic? That’s lights 2.0.
Brad Hine:
I like that. I’m going to reuse that.
Lee Ayers:
It really is. It’s lights 2.0.
We needed electric light. We did a video for our company for our 75th anniversary, I guess it was. Anyway, it was a significant anniversary, and we found a few of the old timers still alive that remembered when their lights were turned on. And they reminisced about how they had survived. They had lived before electric service, but life was hard. Life was tough. Well today, life without the internet, without access to the internet is tough. It’s hard. There are things we just can’t do.
Brad Hine:
As we found out in the last three years, right?
Lee Ayers:
That’s right. The pandemic that forced us all to stay home for some period at some level taught us that. And the world changed. How things are delivered to us changed. They’re now delivered by the internet, and we’ve got to be connected to it to even receive the service. You manage doctor’s appointments over the Internet, all kinds of things. It’s a new utility. It’s lights 2.0. This is what the public needs.
Brad Hine:
In your project, did you get everything in before 2020? How did that work out?
Lee Ayers:
We were about 75% overhead facilities and about 25% of the line miles are buried underground facilities. Out of 60,000 meters, it’s about 35,000 overhead, 25,000 underground. We had the overhead finished in the fall of 2019.
Brad Hine:
Now that’s visionary planning.
Lee Ayers:
No, that’s just dumb luck. But we were taking a brief pause to reorganize ourselves to build out in our existing underground subdivisions.
At least we thought it would be a brief pause. But then March of 2020 rolled around, folks realized they had to work from home, school from home, do from home, and live from home. And the orders for internet connection rolled in.
Brad Hine:
So game on.
Lee Ayers:
Yeah, we stayed very busy for 12 to 18 months. I mean, we’re still busy, don’t get me wrong. But, I mean, we could barely keep up with new installations during that period. Which gave us a little better reset and led us to develop better plans for how we were going to go into these underground subdivisions. We’re still hooking them up every day, every week, still working hard to get it out there. We’ve got close to 18,000 installed drops for homes and businesses. And we’re down to about 9,000 homes left to build the basic facility to so we can then install drops. So we’re closing in on it.
Brad Hine:
That’s an amazing accomplishment for your community. We talk a lot about the mission of ECs like yourselves and try to connect folks. And obviously the interconnection of electric, smart grid. You and I talked a little bit about broadband infrastructure and ISP services just being a piece of that. Thanks for sharing your journey by the way. I know you had a session this morning on the journey to multi-gig.
Lee Ayers:
Yeah. I’m the UTCs broadband committee chair. And one of the things that we try to add clarity to is that broadband and high-speed internet, though very closely related, are subtly different. To us, particularly at Mid-Carolina and to many of the UTC members, broadband is an infrastructure. That’s that fiber optic cable plant we put out there.
High-speed internet is just a service that’s delivered over that infrastructure. There are already other services, a lot of services that help us run the power company. I won’t go into all those details, but that already exists — home security and video. Yeah, it’s transported via the Internet, but it’s yet another service. And as broadband infrastructures are developed, more services are going to be created that can be transported and delivered over high-quality broadband infrastructures.
When we rolled out our fiber-to-the-home program, the only distribution equipment for high-speed internet equipment operated at top speed of one gig. Well today, top speed equipment is readily available and basically the standard offering at 10 gigs.
We get the question, “Well, how much speed does somebody need?”
Well, our position is, it’s not a question of how much they need. The question is, how much did he want? If you want it and are willing to pay the bill for it, we need to be ready to provide it.
As I told them in the presentation this morning, my theory is kind of the top gun effect, “I feel the need, the need for speed.” If you feel the need for speed, let me sell it to you.
Brad Hine:
It’s a great ’80s reference. Thank you for that.
Lee Ayers:
We dated ourselves a little with that reference. But particularly when businesses, when industry approaches us about providing internet service to their business, their manufacturing concern, whatever it may be, they’ll ask the question, “What speed do you offer?”
We say, “What speed would you like?” And they look at us funny and we go, “No, really. I mean, tell us what speed you’d like.”
Because in all reality, if you can say you want that speed, there’s a way that we can provide it. Now there’s a cost. We may have to upgrade some equipment, but the fiber optic broadband infrastructure underlying it will support the speeds that the electronics manufacturers create. It will support those speeds for equipment generations to come, 10 gigs, 25 gigs, 50 gigs, 100 gigs, and so on. We are nowhere near figuring out how fast and how much data we can move over laser light over fiber optics.
Brad Hine:
Right. If that fiber pipe is there, you’re just turning it on. If it needs to go to 10 gigs for a customer that needs that, you’ll provide that. One gig for subscriber services. I always wanted to know from the folks that handle my fiber and internet services, I just don’t want it ever to go below 50 or don’t go below 100. They keep telling me how much they’ll give me. I just don’t want it to go below a certain threshold because then I know I’m in trouble and latency happens throughout the house.
Lee Ayers:
Right. And you figured out where your latency, your point of sensitivity to latency is for your home. It’s different for everybody. And you’re right. It really is latency. And that’s how long the little rotating arrow spins on the video screen.
Yeah, it is buffering. And sometimes it’s just response time. Sometimes the latency issue is even more subtle and less obvious than seeing that buffering going on. I’ll share a quick little story.
Our CEO had our service installed at his home once we got built by his house and initially installed it gigabit. After a few months, he said, “Well, I think we could get by with slower.” And it really wasn’t that it was all that expensive, but he’s a thrifty individual. He said, “I’ll save $20 a month. Turn me back to 500.”
And I think he did that on a Friday. He told the internet service people to back him down to the 500 MB tier. They did. He got home that evening, and his grandson was over. He walked in and his grandson goes, “Pop, Pop, something’s wrong with the internet.”
And Bob said, well, “What’s wrong with it?”
He said, “It’s slow. All my Fortnite soldiers are getting killed. It’s slowed down. It’s too slow. I usually win. I can’t win.”
Needless to say, Monday morning, he went back to the gigabit service so his grandson can continue to be a Fortnite champion.
Brad Hine:
Lee, that’s another first for this show. Now we’ve had a Fortnite reference on a Broadband Bunch episode. Thank you for that. That’s great. Gaming’s huge, right?
Lee Ayers:
Well, it is. Cloud-spaced gaming is all about response time. I don’t game, but it’s who gets the shot off first, who picks up the treasure first. It’s all about who can get the fastest response. A lot of times it’s not just your download speed, but it’s your upload speed.
And the beauty of almost all fiber optic delivery systems for broadband is they work very well in a symmetrical format, 100 MB down, 100 MB up. Whereas a lot of the other media, coax, twisted pair copper, et cetera, generally have faster download than upload. But yeah, many of our requests for gigabit are based on gamers in the home, and all gamers aren’t teenage grandchildren. They come in all ages.
But as we see things going, that’s where we’re heading. All our new front-end equipment is now. We don’t install anything but 10 gigs because that is what’s readily available. That’s where the marketplace is moving to.
Brad Hine:
Man, that’s moved quickly. I know you’ve been involved on the broadband committee at UTC for such a long time. Are there any surprises for you this year maybe in terms of content, in terms of technological innovation, or things like that?
Lee Ayers:
I don’t know that it’s so much a surprise but it’s more of a welcome revelation. As I got started with UTC and the broadband space for UTC, it was mostly electric co-ops talking about it. And a lot of it’s due to the availability of federal funding, grant programs, and whatnot. Other utilities, IOUs, municipals, and public power utilities, some of them state laws won’t allow them to be service providers. But laws are changed. Funding’s changed; things have changed. And I see a lot more of the folks that work for those utilities looking for information and looking to see how they can help their communities.
They’ve got their other profit motives and other reasons for their business, but ubiquitous broadband infrastructure is now accepted as a utility need. It’s a service people need.
Brad Hine:
Right. Absolutely. I’ve told this example many times. Living in Atlanta, and you’re in South Carolina, I’m sure maybe it’s the same as the temperatures dip in the winter. It’s cold; it’s not really cold. But if your gas goes off, there’s no heat in your house, you put a coat on. You put a fire in the fireplace if it doesn’t require gas, and you wait half a day or a day for someone to show up and fix the gas. No big deal.
If we are without broadband now for 15 minutes, everyone in my house will come to me to fix it. And if I’m not at home, which sometimes I’m at a conference, I’m the IT guy. So they’re screaming through the phone and texting me 15 minutes after the broadband’s down. So it’s being reprioritized on our utility list for sure.
Lee Ayers:
It’s being reprioritized by the users. We’re also having to evaluate our prioritization in restoration. When we first entered this business in 2016, we said, “Oh, well sure, we’ll worry about getting the power back up. Then we’ll worry about getting the fiber back up.”
And thank goodness we hadn’t had any big really bad storms, but some one-day event kind of things.
Really and truly, we are now getting people to say, “You can leave my power off, get my internet back up.” I mean, think about it. They’ve got a battery backup, or they’ve got a little generator they can run their router off of. They can go to the car and charge their phone, charge their iPad, charge their tablet, or whatever. And they’re happy and occupied and not worried about the electricity nearly as bad if their internet’s up. Who would have thought?
Brad Hine:
Absolutely. Well, as we wind down another episode of the Broadband Bunch, I want to thank you for joining us, Lee. Thank you for all the work you’ve done participating in the UTC on the broadband committee.
I ask a question of most folks on the Broadband Bunch. I call it the Back to the Future question. Just like the movie, if you could get in your time machine DeLorean and go back to when you started in this industry or started at MCEC, is there some advice you could whisper in your own ear to give you a leg up or a heads-up on what you were about to embark on?
Lee Ayers:
A couple of pieces. Yeah. And one of those pieces is to always build for the future. Always design and build your facilities for the long haul. The analog microwave system I said we put in; we didn’t add any extra capacity. We were using proven technology. Well, in today’s world, in the technology evolution cycle, if it’s been proven, is also obsolete. And that’s how fast technology evolves today.
So plan for the future, build for the future. And the one thing we’ve said at Mid-Carolina over and over is if we had only seen this coming when we were worried about cable TV and how they were attaching to our poles and getting in our way, we’d have been putting up cable ourselves.
In 2005, if we could have had the thought, the vision, the courage to take on the risk and do it. But I want to emphasize it’s still not too late. Whenever you get there, you will wish you had done it earlier.
Brad Hine:
Right. Well thank you so much for being on the Broadband Bunch today, Lee.
Lee Ayers:
Thanks for the invitation, Brad.
Brad Hine:
Absolutely. And I’ll invite you back in the future. I know Brett told me a little earlier, Brett Kilborn, that there are nine regionals for UTC. So I know our paths will cross at some point again very soon.
Lee Ayers:
That’s right. And my co-op, Mid-Carolina, is part of UTC region three. We’ll be meeting the first week in October in Knoxville, Tennessee. So come on see us.
Brad Hine:
Excellent. I would love to. Thank you so much.
From everyone at the Broadband Bunch and our listeners today, we will talk again soon. Thanks a bunch, everyone.
