The answer, unfortunately, is generally no.
Not only can you not custom-cut a star shape for your book about stars, but in general you can't even decide you want your book to be 10.725 inches wide by 3.2953 inches high. You can't set the size. The companies have machines that work with certain set sizes. They provide that list of set sizes to you. You choose one and go forward. Even for a certain book size, they also have a minimum and maximum number of pages you can use. If you want to make a standard 8x10 book, you can't make it with 4,000 pages. That just isn't possible.
This is one of those tradeoffs you make when you're working with self publishing. They have a machine. You're using their machine. You have to work with what that machine is capable of creating.
What do you do if you desperately want to make a book about the planet Mars which is a round shape? I'm afraid you'll have to directly contact an actual printing company. Nearly all of those will require for you to make an up-front order of 500 or more copies for them to justify creating the setup to make your special shape. That can easily run into the thousands of dollars.
Or, conversely, you can lobby with traditional publishers to take on your project so they pay all those costs for you.
Either way, it's important to know that those custom-sized jobs require a fair amount of effort. Someone will be paying for that - whether it's you or the publishing company.
Lisa Shea's Editing Services
Lisa Shea Free Ebooks
Lisa Shea Full Library of Published Books
Getting Your Book Published
Writing Tips and Online Books
Lisa Shea Medieval Romance Novels
Online Literary Magazines
Lisa Shea's Homepage